Re: PG_DUMP very slow because of STDOUT ??

Hi Craig,

Now I did the test with top too. Free RAM is around 900. And there doesn't seem to be other processes eating memory away (looked at it with top/htop). The other procs having more RAM did have it before (mostly some postgres processes), and don't grow it at an exporbitant rate. One coul see with free, that most of the time, when it is eaten away, it goes directly to cache. I also observe, that the lower "free" is, the slower COPY-to-STDOUT becomes ... and linearly, the number of times one can see congestion_wait in PS grows. So, with more "free" available (even when it is not eaten away anymore ...) the more responsive COPY-to-STDOUT.

Well, I have now also rerun a few times my plain COPY-to-FILE test. That one seems to behave similarly with the RAM (when monitored via "free -m") ... but it never gets into congestion. So, maybe, just maybe ... could this be some kind of race condition? I mean, the file cache going for RAM (or the memory mapped file ???) and the SOCKET going for some memory too, and then they somehow overwhelm the memory manager (which shouldn't happen). ... Well, this might be a very wild speculation, but still.

I now also looked at the stack, before the congestion_wait. There are some "interesting" calls leading to it. Like "shrink_list", "zone_reclaim" ... after the call of "get_page_from_freelist". Sounds like some heavy memory juggling going on (to get new free pages) ... What ever the reason is for this.

Andras Fabian

-----Ursprüngliche Nachricht-----
Von: Craig Ringer [mailto:craig(at)postnewspapers(dot)com(dot)au]
Gesendet: Dienstag, 13. Juli 2010 14:11
An: Andras Fabian
Cc: pgsql-general(at)postgresql(dot)org
Betreff: Re: AW: AW: AW: AW: [GENERAL] PG_DUMP very slow because of STDOUT ??

On 13/07/10 18:57, Andras Fabian wrote:
> OK, so here I should - maybe - look around the sockets. Hmm. Well, in the case of my experiments we are talking about Unix sockets, as I am only connecting locally to the server (not real networking involved). Are there any ideas, where such a Unix Socket could impose such extreme buffering ??? And can/could I control its behavior at all?? (or would it be a thing, which can only be controlled from C-Code ... which would fall back to PostgreSQL as the initiator).

Confirm your issue is actually the socket between client and server first.

Are you sure the client isn't buffering the data? Or a pager like
"less"? Or terminal scrollback?

Run "top" and sort by memory use by pressing Shift-M. Run your query.
Look at the output and see if anything grows lots.

You really need to start looking at if this is memory consumption, and
what's using it. Don't get fixated on one answer until you've excluded
other possibilities.

--
Craig Ringer