Re: Backends dying due to memory exhaustion--I'm stonkered

Tom Lane <tgl(at)sss(dot)pgh(dot)pa(dot)us> writes:

> If you want to force things to happen then you should send SIGINT to
> the postmaster, which in turn will SIGTERM its backends, which in
> theory will abort their transactions and shut down. (No, this isn't
> real consistent, but we wanted the right things to happen when a
> system-wide shutdown sends SIGTERM to all the processes. Usually
> you shouldn't be manually killing individual backends anyway, so the
> lack of consistency in signal meaning shouldn't mean much...)

OK, finally got another crash. There is no core file, and all the
backends exited cleanly when I sent SIGINT to the postmaster.

I realized something very interesting that I hadn't twigged to before:
OpenACS was running a scheduled notification (that involves least one
update) at the exact same time that VACUUM was running on the
database. This time, I got a different memory-related error in the
VACUUM:

basename: error in loading shared libraries: libc.so.6: failed to map segment from shared object: Cannot allocate memory
/usr/local/pgsql/bin/vacuumdb: /usr/local/pgsql/bin/vacuumdbpsql: No such file or directory
Vacuuming template1
VACUUM
[...]

This is *very* interesting, as it indicates that the system as a whole
is running out of memory right as the VACUUM and OpenACS notification
thread kick off.

What may be happening is that the update that ACS wants to do is
blocked by the VACUUM, and something in there is going nuts and eating
memory like crazy.

The only fly in this ointment is that the above log fragment seems to
indicate that the system ran out of memory before VACUUM even started
(the 'basename' command in 'vacuumdb' is what crapped out AFAICS).

So maybe the AOLServer/ACS combo is the culprit here.

I've moved the VACUUM to a time that doesn't coincide with the ACS
notification stuff; I'll see if we continue to get crashes, and when.

Anyhow, thanks for listening to my ramblings; looks like PG may be
off the hook, which would make me very happy, as I think it's a great
product.

-Doug