Re: Postmaster Out of Memory

Tom Lane wrote:
> I was sort of expecting you to come back and say that you
> thought the process might have done 640K TRUNCATEs over its lifespan,
> but I guess not?

That's possible. The process does twelve TRUNCATEs every minute. The
problem we're talking about seems to occur only when the system has been
running for quite some time. Still, that's seems like an awefully slow
rate. Is it really likely that a postmaster backend would accumulate
relcache entries in this case?

> What about temporary tables?

We don't use AS TEMP tables, but we have something like thirty tables
that we create, use, drop and eventually re-create. This gets done
about twice a day or so.

> (Note: you have the list
> of tables involved, in the form of that dump you showed --- does
> eyeballing the list suggest any pattern to you? Are there duplicate
> table names in the list?)

What stands out is that nearly all of the entries seem to be pg_*
tables. A "grep '^pg_' crash_dump | wc -l" gives 640499 entries, which
is pretty nearly the entire file. We are careful not to name our tables
with the "pg_" prefix, so I believe these are system tables. This is
part of what originally made us believe pg_autovacuum was responsible.

> Given that you are showing "SHARE" as 180M, I think
> it's pretty likely you've got only 2MB of actual local storage in that
> backend.

Ah, good point. I was assuming that memory was being allocated from a
shared pool, but I have no evidence to support that.

> If it's a true leak, then letting it run until RSS is well beyond shared
> memory will prove it.

We'll give that a try over the weekend and see where it goes. (We have
since stopped the script because Joe is improving the instrumentation of
the code so that we don't overflow the logs again.)

Jeff