From: | Heikki Linnakangas <hlinnakangas(at)vmware(dot)com> |
---|---|
To: | Greg Stark <stark(at)mit(dot)edu>, Andres Freund <andres(at)2ndquadrant(dot)com> |
Cc: | Maciek Sakrejda <m(dot)sakrejda(at)gmail(dot)com>, PostgreSQL Bugs <pgsql-bugs(at)postgresql(dot)org> |
Subject: | Re: BUG #10432: failed to re-find parent key in index |
Date: | 2014-06-04 12:06:19 |
Message-ID: | 538F0BBB.5000308@vmware.com |
Views: | Raw Message | Whole Thread | Download mbox | Resend email |
Thread: | |
Lists: | pgsql-bugs |
On 06/04/2014 02:14 PM, Greg Stark wrote:
> Ok, I made some progress. It turns out this was a pre-existing problem
> in the master. They've been getting "failed to re-find parent" errors
> for weeks. Far longer than I have any WAL or backups for.
Bummer. You don't have logs reaching back to when the first error
happened either, I presume.
The most likely cause for originally failing to insert a downlink is
running out of disk space when trying to split the parent page. (Even
that's pretty unlikely, though.)
> What I did find that was interesting is that this error basically made
> the backups worthless. I could build a hot standby and connect and
> query it. But as soon as recovery finished it would try to clean up
> the incomplete split and fail. Because it had noticed the incomplete
> split it had skipped every restartpoint and the next time I tried to
> start it it insisted on restarting recovery from the beginning. If we
> had been lucky enough not to do any page splits in the broken index
> while the backup was being taken all would have been fine. But that
> doesn't seem to have happened so all the backups were unrecoverable.
It's only *incomplete* splits that happen while the backup is taken that
matter. i.e. a page is split but inserting the downlink to the parent
fails for some reason. If a page is split and the downlink is inserted
successfully, that's OK.
> So a few thoughts on how to improve things:
>
> 1) Failed to re-find parent should perhaps not be FATAL to recovery.
> In fact any index replay error would really be nice not to have to
> crash on. All crashing does is prevent the user from being able to
> bring up their database and REINDEX the btree. This may be another use
> case for the machinery that would protect against corrupt hash indexes
> or user-defined indexes -- if we could mark the index invalid and
> proceed (perhaps ignoring subsequent records for it) that would be
> great.
Yeah, that would be nice as an option.
> 2) When we see an abort record we could check for any cleanup actions
> triggered by that transaction and run them right away. I think the
> checkpoints (and maybe hot standby snapshots or vacuum cleanup
> records?) also include information about the oldest xid running, they
> would also let us prune the cleanup actions sooner. That would at
> least find the error sooner. In conjunction with (1) it would also
> mean subsequent restartpoints would be effective instead of
> suppressing restartpoints right to the end of recovery.
You can't do any cleanup actions until you've recovered all the WAL. A
cleanup action means inserting a tuple to the parent page, and you can't
do that in the middle of recovery.
But we could detect the case and give a warning.
> 3) The lack of logs around an error during recovery makes it hard to
> decipher what's going on. It would be nice to see "Beginning Xlog
> cleanup (1 incomplete splits to replay)" and when it crashed "Last
> safe point to restart recovery is 324/ABCDEF". As it was it was a
> pretty big mystery why the database crashed, the logs made it appear
> as if it had started up fine. And it was unclear why restarting it
> caused it to replay from the beginning, I thought maybe something was
> wrong with our scripts
Yeah. All of this is gone in 9.4 anyway, but if you can come up with
something non-invasive that can be back-patched...
- Heikki
From | Date | Subject | |
---|---|---|---|
Next Message | Vik Fearing | 2014-06-04 13:47:42 | Re: BUG #10526: bad sequence rename |
Previous Message | Andres Freund | 2014-06-04 11:35:19 | Re: BUG #10432: failed to re-find parent key in index |