Re: postmaster deadlock while logging after syslogger exited

On Fri, Nov 17, 2017 at 10:50 AM, Andres Freund <andres(at)anarazel(dot)de> wrote:
> On 2017-11-06 15:35:03 -0500, Tom Lane wrote:
>> David Pacheco <dap(at)joyent(dot)com> writes:
>> > I ran into what appears to be a deadlock in the logging subsystem. It
>> > looks like what happened was that the syslogger process exited because it
>> > ran out of memory. But before the postmaster got a chance to handle the
>> > SIGCLD to restart it, it handled a SIGUSR1 to start an autovacuum worker.
>> > That also failed, and the postmaster went to log a message about it, but
>> > it's blocked on the pipe that's normally connected to the syslogger,
>> > presumably because the pipe is full because the syslogger is gone and
>> > hasn't read from it.
>>
>> Ugh.
>
> I'm somewhat inclined to say that one has to live with this if the
> system is so resource constrainted that processes barely using memory
> get killed.
>
> We could work around a situation like that if we made postmaster use a
> *different* pipe as stderr than the one we're handing to normal
> backends. If postmaster created a new pipe and closed the read end
> whenever forking a syslogger, we should get EPIPEs when writing after
> syslogger died and could fall back to proper stderr or such.

I don't have the code on top of my mind, but isn't a custom fd causing
a small penalty when redirection_done is switched to true because the
first process generating a message to the syslogger pipe needs to open
it first if not done yet? So you'd need proper locking to save from
race conditions. Or is the first message redirected message always
generated by the postmaster or the syslogger? I don't recall that this
is actually true..
--
Michael