Re: corruption issue after server crash - ERROR: unexpected chunk number 0

Mike Broers <mbroers(at)gmail(dot)com> wrote:

> Hello we are running postgres 9.2.5 on RHEL6, our production
> server crashed hard and when it came back up our logs were
> flooded with:

> ERROR:  unexpected chunk number 0 (expected 1) for toast value 117927127 in pg_toast_19122

Your database is corrupted.  Unless you were running with fsync =
off or full_page_writes = off, that should not happen.  It is
likely to be caused by a hardware problem (bad RAM, a bad disk
drive, or network problems if your storage is across a network).

If it were me, I would stop the database service and copy the full
data directory tree.

http://wiki.postgresql.org/wiki/Corruption

If fsync or full_page_writes were off, your best bet is probably to
go to your backup.  If you don't go to a backup, you should try to
get to a point where you can run pg_dump, and dump and load to a
freshly initdb'd cluster.

If fsync and full_page_writes were both on, you should run hardware
diagnostics at your earliest opportunity.  When hardware starts to
fail, the first episode is rarely the last or the most severe.

--
Kevin Grittner
EDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company