Re: Streaming replication slave crash

On Mon, 2013-09-09 at 13:04 -0700, Mahlon E. Smith wrote:
> After some wild googlin' "research", I saw the index visibility map fix
> for 9.2.1. We did pg_upgrade in-between versions, but just to be sure I
> wasn't somehow carrying corrupt data across versions (?), I went ahead
> and VACUUMed everythng with the vacuum_freeze_table_age set to 0, and
> went on with my life, hoping I had removed whatever demons were running
> around in there.

You may have seen only partial information about that bug and the fix.
See the first item in the release notes here:

http://www.postgresql.org/docs/current/static/release-9-2-1.html

And the actual fix here:

http://git.postgresql.org/gitweb/?p=postgresql.git;a=commit;h=beb850e1d873f8920a78b9b9ee27e9f87c95592f

To rid the daemons entirely, after upgrading to 9.1+, you must REINDEX
all btree and GIN indexes that may have been touched by 9.2.0. Since the
actual problem you see is on an index (as you say later in your report),
then I would suggest that you do that.

Not 100% sure this is the root cause of your problem, of course, but the
symptoms seem to line up.

> Unlike Quentin's original message, simply restarting the slave didn't
> bring it back to life. I had to pg_start_backup/rsync again from the
> master, at which point:

It looks OK to me, so I think you are in the clear.

If you are particularly unlucky, your master server crashed (while still
on 9.2.0) without writing the data and left your master copy of the
index corrupt. If you are worried about that, you can do another re-sync
after you finish the REINDEXing. This is not necessary unless you
experienced at least one crash on 9.2.0.

Regards,
Jeff Davis