Re: CRITICAL HELP NEEDED! DEAD DB!

Cott Lang <cott(at)internetstaff(dot)com> writes:
> Sep 24 10:22:37 snafu postgres[18306]: [2-1] LOG: database system was
> interrupted while in recovery at 2004-09-24 10:21:41 MST
> Sep 24 10:22:37 snafu postgres[18306]: [2-2] HINT: This probably means
> that some data is corrupted and you will have to use the last backup for
> recovery.
> Sep 24 10:22:37 snafu postgres[18306]: [3-1] LOG: checkpoint record is
> at 9A/C2022368
> Sep 24 10:22:37 snafu postgres[18306]: [4-1] LOG: redo record is at
> 9A/C2022368; undo record is at 0/0; shutdown FALSE
> Sep 24 10:22:37 snafu postgres[18306]: [5-1] LOG: next transaction ID:
> 197841225; next OID: 715436086
> Sep 24 10:22:37 snafu postgres[18306]: [6-1] LOG: database system was
> not properly shut down; automatic recovery in progress
> Sep 24 10:22:37 snafu postgres[18306]: [7-1] LOG: redo starts at
> 9A/C20223B0
> Sep 24 10:22:37 snafu postgres[18306]: [8-1] PANIC: btree_insert_redo:
> failed to add item
> Sep 24 10:22:37 snafu postgres[18299]: [2-1] LOG: startup process (PID
> 18306) was terminated by signal 6
> Sep 24 10:22:37 snafu postgres[18299]: [3-1] LOG: aborting startup due
> to startup process failure

> Any suggestions to recover?! I'm dead in the water! Please!!!

I think your only chance is pg_resetxlog. Be aware that you won't
necessarily have a consistent database afterwards --- in particular,
whichever index that failure is about is certainly broken. I'd
recommend a dump and reload, plus as much manual verification of data
consistency as you can manage.

How did you get into this state, anyway?

regards, tom lane