Re: [GENERAL] postmaster deadlock while logging after syslogger exited

Hi,

On 2017-11-20 10:13:57 -0800, David Pacheco wrote:
> I expect what happened is that the syslogger process attempted to allocate
> memory, failed because the system was low, and explicitly exited. That's
> consistent with an exited process, no core file generated, and the "FATAL"
> "out
> of memory" entries in the log I posted.

Well unfortunately we're presumably not going to see output from
syslogger's death itself, chicken and egg etc.

> Tom Lane wrote:
> > Hm, so that's another angle David didn't report on: is it possible that
> > his workload could have resulted in a very large volume of incomplete
> > in-progress log messages?
>
> Yes. I mentioned in my Nov 6 mail that large log messages over a
> short period appear to have been a major contributing factor:

So it's actually quite possible that syslogger did use a fair amount of
memory. Could you show its memory usage, just via ps?

> This is a critical point. As far as I can tell, all that's necessary
> for this deadlock to occur is:
>
> - the syslogger is unable to make forward progress (e.g., because it
> segfaulted)

This specific case I don't care that much about, but ...

> - enough other processes write to the pipe that it fills up before the
> postmaster can restart the syslogger
> - the postmaster attempts to write to the log as part of any of its signal
> handling operations
>
> It seems to me that the same thing can happen if there were a bug in the
> syslogger that resulted in a segfault around the same time that the
> postmaster
> attempted to start an autovacuum worker, for example (if there was also
> reasonably high log traffic).

Greetings,

Andres Freund