Re: warning message in standby

Heikki Linnakangas wrote:
> On 12/06/10 04:19, Bruce Momjian wrote:
> > Robert Haas wrote:
> >>> If my streaming replication stops working, I want to know about it as
> >>> soon as possible. WARNING just doesn't cut it.
> >>>
> >>> This needs some better thought.
> >>>
> >>> If we PANIC, then surely it will PANIC again when we restart unless we
> >>> do something. So we can't do that. But we need to do something better
> >>> than
> >>>
> >>> WARNING there is a bug that will likely cause major data loss
> >>> HINT you'll be sacked if you miss this message
> >>
> >> +1. I was making this same argument (less eloquently) upthread.
> >> I particularly like the errhint().
> >
> > I am wondering what action would be most likely to get the
> > administrator's attention.
>
> I've committed the patch to disconnect the SR connection in that case.
> If the message needs improvement, let's do that separately once we
> figure out what to do.
>
> Seems like we need something like WARNING that doesn't cause the process
> to die, but more alarming like ERROR/FATAL/PANIC. Or maybe just adding a
> hint to the warning will do. How about
>
> WARNING: invalid record length at 0/4005330
> HINT: An invalid record was streamed from master. That can be a sign of
> corruption in the master, or inconsistency between master and standby
> state. The record will be re-fetched, but that is unlikely to fix the
> problem. You may have to restore standby from base backup.

I am thinking about log monitoring tools like Nagios. I am afraid
they are never going to pick up something tagged WARNING, no matter
what the wording is. Crazy idea, but can we force a fatal error line
into the logs with something like "WARNING ...\nFATAL: ...".

--
Bruce Momjian <bruce(at)momjian(dot)us> http://momjian.us
EnterpriseDB http://enterprisedb.com

+ None of us is going to be here forever. +