Re: Database Corruption - last chance recovery options?

Thomas F. O'Connell wrote:
>
> On Jan 4, 2007, at 11:24 PM, Michael Best wrote:
>
>> When I finally got the error report in the morning the database was in
>> this state:
>>
>> $ psql dbname
>>
>> dbname=# \dt
>> ERROR: cache lookup failed for relation 20884
>
> Do you have your error logs, and were there any relevant errors in them
> preceding your inability to get a table listing via psql?

From pgstartup.log I have the following, but I think these were from
when I actually restarted the database the next day after finding out
about the corruption.

FATAL: could not create shared memory segment: Invalid argument
DETAIL: Failed system call was shmget(key=5432001, size=86441984, 03600).

Set your memory requirement too high in postgresql.conf, reload instead
of restarting the database, it silently fails sometime later? Just a
theory, as far as I know those errors were probably just from failed
startup attempts when I was rebooting the server after seeing the data
corruption.

From Wednesday when the problems started occuring I have a SIGHUP which
I assume is the reload, and then a number of errors such as:
LOG: received SIGHUP, reloading configuration files
LOG: transaction ID wrap limit is 1074563238, limited by database
"template1"

The last line is repeated 34 times before I start seeing things like
ERROR: relation "auth_users" does not exist

On Thursday after the corruption there are more cache lookup failed log
entries and more relation does not exist errors as well as some lines
that say "dbname" instead of "template1":

LOG: transaction ID wrap limit is 1074563466, limited by database "dbname"

Interspersed with all these entries are NOTICES that are from ongoing
development on a different database on the same server (I should talk to
them about insanely long names): 236 of these NOTICES on the day of the
problems and about 638 since last Saturday.

NOTICE: identifier
"docmanager_document__document_type__workflow__workflowstatehasaction"
will be truncated to
"docmanager_document__document_type__workflow__workflowstatehasa"
NOTICE: identifier
"m2m_docmanager_documenttype__workflow__workflowstatehasaction__permitted_
roles" will be truncated to
"m2m_docmanager_documenttype__workflow__workflowstatehasaction__"

That's all I have. No timestamps on the logs means I have to guess as
some of the timing as well.

-Mike