Re: The database system is in recovery mode

Tom Lane wrote:

> "Iavor Raytchev" <pobox(at)verysmall(dot)org> writes:
> > Before the crash is this one -
> > FATAL 1: Sorry, too many clients already
>
> That should be harmless --- I doubt it's relevant to the crash, unless
> you have timestamps that prove it happened just before the crash.

True, after the recovery we had this several times without a crash.

> > IpcSemaphoreLock: semop(id=-1) failed: Invalid argument
>
> This is a new one on me. AFAICT it must mean that something clobbered
> the PROC array in shared memory (the semId argument to IpcSemaphoreLock
> is always taken from MyProc->sem.semId). Never heard of that happening
> before.
>
> > DEBUG: pq_flush: send() failed: Bad file descriptor
>
> And that seems odd too; it suggests something overwrote MyProcPort->sock,
> which is another un-heard-of failure mode.
>
> > Shall we simply restart?
>
> Yeah, I'd try that, but I'd also suggest looking for system-wide
> problems. Normally the postmaster can recover by itself from any sort
> of failure in child processes ... the fact that it didn't seem to do so
> is another strikingly odd behavior. I'm starting to wonder about
> possible hardware flakiness. Bad RAM maybe?

There was something strange - one (old?) db connection that refused to be
killed. After we managed to shut down the postmaster (not sure anymore, but
I think we had to kill it) - this db connection could be killed as well.
After new start of the postmaster all was OK. Seems there has been something
stuck that has been old and now it was just triggered... We still
investigate the surrounding sofware and hardware.

Thanks for the help.

Iavor