Re: ERROR: missing chunk number 0 for toast value 382548694 in pg_toast_847386

Hi Vibhor,

>> Did you find anything suspicious in pg_log?

We've been through it all and did not see anything we didn't expect.

>> Please share recovery.conf information.

We did interrupt the restore a few times. The initial recovery.conf file contained only:

restore_command = 'gunzip -c /mnt/dbsbackup/pg_xlog/%f.gz > %p'

Later we decided to replace the recovery command with a wrapper script that would allow us to leave the restore going unattended over the weekend, and complete up until the latest WAL file on the original database (which is still running). We changed the recovery command to:

restore_command = '/var/lib/pgsql/data/db_restore_dm %f %p'

where the script db_restore_dm contained:

#!/usr/bin/perl

use strict;

my ($pg_f, $pg_p) = @ARGV;
exit 1 if $pg_f eq '00000001.history';

my $xlogBackupFile = "/mnt/dbsbackup/pg_xlog/$pg_f.gz";

while (! -f $xlogBackupFile and !$triggered) {
sleep 2;
}

while (1) {
system("gunzip -c $xlogBackupFile > $pg_p");
last if ($? >> 8 == 0);
sleep 2;
}

We were concerned that shutting down / starting up while recovery is ongoing might cause some problems, but the pg documentation indicates this should be OK and we saw no cause for concern in the pg logs.

> Did you verify base backup (Restoring without PITR) ?

I guess you mean did we restore it up until consistency reached? No so far we have continued restoration until we reached the last WAL file made by the original database.

Regards // Mike

-----Original Message-----
From: Vibhor Kumar [mailto:vibhor(dot)kumar(at)enterprisedb(dot)com]
Sent: Monday, 28 February 2011 3:25 PM
To: Michael Harris
Cc: pgsql-general(at)postgresql(dot)org
Subject: Re: [GENERAL] ERROR: missing chunk number 0 for toast value 382548694 in pg_toast_847386

On Feb 28, 2011, at 9:46 AM, Michael Harris wrote:

> The main reason we have not gone down that path that after getting this error we do not have any confidence in the integrity of the rest of the database after performing the restore - maybe there are many more tables with corruption in them.
May be or may not be.

Did you find anything suspicious in pg_log?
Please share recovery.conf information.

> We are mainly concerned that our backups are valid. The fault does not seem to be present on the original database, only after restoring from the base backup + PITR.

Did you verify base backup (Restoring without PITR) ?

Thanks & Regards,
Vibhor Kumar
vibhor(dot)kumar(at)enterprisedb(dot)com
Blog:http://vibhork.blogspot.com