From: | Yang Zhang <yanghatespam(at)gmail(dot)com> |
---|---|
To: | Scott Marlowe <scott(dot)marlowe(at)gmail(dot)com> |
Cc: | pgsql-general(at)postgresql(dot)org |
Subject: | Re: Sorting performance vs. MySQL |
Date: | 2010-02-23 07:02:33 |
Message-ID: | 9066fa251002222302y31e1992awdddfa147d11ad9cf@mail.gmail.com |
Views: | Raw Message | Whole Thread | Download mbox | Resend email |
Thread: | |
Lists: | pgsql-general |
On Tue, Feb 23, 2010 at 1:48 AM, Scott Marlowe <scott(dot)marlowe(at)gmail(dot)com> wrote:
> On Mon, Feb 22, 2010 at 10:51 PM, Yang Zhang <yanghatespam(at)gmail(dot)com> wrote:
>> nnnnnOn Mon, Feb 22, 2010 at 3:45 PM, Scott Marlowe
>> <scott(dot)marlowe(at)gmail(dot)com> wrote:
>>>
>>> What do things like vmstat 10 say while the query is running on each
>>> db? First time, second time, things like that.
>>
>> Awesome -- this actually led me to discover the problem.
>>
>> vmstat showed no swapping-out for a while, and then suddenly it
>> started spilling a lot. Checking psql's memory stats showed that it
>> was huge -- apparently, it's trying to store its full result set in
>> memory. As soon as I added a LIMIT 10000, everything worked
>> beautifully and finished in 4m (I verified that the planner was still
>> issuing a Sort).
>>
>> I'm relieved that Postgresql itself does not, in fact, suck, but
>> slightly disappointed in the behavior of psql. I suppose it needs to
>> buffer everything in memory to properly format its tabular output,
>> among other possible reasons I could imagine.
>
> It's best when working with big sets to do so with a cursor and fetch
> a few thousand rows at a time. It's how we handle really big sets at
> work and it works like a charm in keeping the client from bogging down
> with a huge memory footprint.
>
Thing is, this is how I got here:
- ran complex query that does SELECT INTO.
- that never terminated, so killed it and tried a simpler SELECT (the
subject of this thread) from psql to see how long that would take.
I.e., my original application doesn't receive the entire dataset.
--
Yang Zhang
http://www.mit.edu/~y_z/
From | Date | Subject | |
---|---|---|---|
Next Message | Wang, Mary Y | 2010-02-23 07:19:34 | How to Output PSQL Errors to a Log File? |
Previous Message | Alex Hunsaker | 2010-02-23 06:49:13 | Re: Sorting performance vs. MySQL |