Database crash in Postgres 8.4.3

From: "Freddie Burgess" <fburgess(at)radiantblue(dot)com>
To: <pgsql-bugs(at)postgresql(dot)org>
Subject: Database crash in Postgres 8.4.3
Date: 2012-08-21 15:58:57
Message-ID: 001e01cd7fb5$df80c9e0$9e825da0$@radiantblue.com
Views: Raw Message | Whole Thread | Download mbox | Resend email
Thread:
Lists: pgsql-bugs

Late yesterday afternoon our DB server went down hard. we tried to re-start
and it went into recovery mode to recover transaction history and failed.
Notable error was:

FATAL: failed to re-find parent key in index "257969064" for split pages
8366/12375

If you look this error up, it indicates issues with the transaction logs and
the inability to recover due to

corrupt or missing transaction logs.

The solution is to:

1. Back up the DB files

2. Run pg_resetxlogs (this might produce corruption due inconsistent data)

3. Dump the DB

4. Reload the DB

Unfortunately this solution is not practical in our case for multiple
reasons.

- We do not have the space for steps 1 & 3

- The time required for steps 1, 3, and 4 is approximately 1 week per step
(3 weeks total) since our database size is approximately 5.4TB

If the data is in an inconsistent state, are there other alternative
solutions, such as finding the index specified in the FATAL error and
somehow dropping it?

and does anyone know what circumstances/conditions might corrupt or cause a
transaction log to go missing.

thanks

Responses

Browse pgsql-bugs by date

  From Date Subject
Next Message Robert Haas 2012-08-21 18:54:28 Re: BUG #4958: Stats collector hung on WaitForMultipleObjectsEx while attempting to recv a datagram
Previous Message Maxim Boguk 2012-08-21 13:30:44 Re: BUG #7500: hot-standby replica crash after an initial rsync