Re: postgres memory management issues?

Richard Yen <dba(at)richyen(dot)com> writes:
> Here is a snippet of my log output (I can give more if necessary):
> Sep 5 18:38:57 tii-db2.oaktown.iparadigms.com Out of Memory: Kill
> process 11696 (postgres) score 1181671 and children.

> My understanding is that if any one postgres process's memory usage,
> plus the shared memory, exceeds the kernel limit of 4GB, then the
> kernel will kill the process off. Is this true?

No. The OOM killer is not about individual process size. It's about
getting the kernel out of the corner it's backed itself into when it's
promised more memory for the total collection of processes than it
can actually deliver. As already noted, fooling with the overcommit
parameter might help, and migrating to a 64-bit kernel might help.
(32-bit kernels can run out of space for "lowmem" long before all of
your 16G is used up.)

ObDigression: The reason the kernel would do such a silly-sounding thing
as promise more memory than it has is that in a lot of cases pages are
shared by more than one process --- in fact, immediately after a fork()
the child process shares *all* pages of its parent --- and it would be
really restrictive to insist on having sufficient RAM+swap for each
process to have an independent copy of shared pages. The problem is
that it's hard to guess whether currently-shared pages will need
multiple copies in future. After a fork() the child's pages are
supposed to be independent of the parent, so if either one scribbles
on a shared page then the kernel has to instantiate separate copies
at that moment (google for "copy on write" for more about this).
The problem is that if there is not enough memory for another copy,
there is no clean API for the kernel to return "out of memory".
It cannot just fail the write instruction, so the only recourse is to
nuke some process or other to release memory. The whole thing makes
considerable sense until you are trying to run critical applications,
and then you just wanna turn it off.

regards, tom lane