Re: BUG #5929: ERROR: found toasted toast chunk for toast value 260340218 in pg_toast_260339342

On 16.03.2011 22:29, Kevin Grittner wrote:
> Tambet Matiisen<tambet(dot)matiisen(at)gmail(dot)com> wrote:
>> Yes, I use pg_dump on live server and the result is
>> rdiff-backupped into development server. Whole SQL dump is 12G
>> without compression and the rdiff delta is about 10-20MB every
>> day. Then I drop pre-live database on development server and
>> recreate it using createdb and psql.
>
> createdb, not initdb? I suggest you backup and delete everything in
> the data directory, and start with initdb, and see whether the
> problem still exists. If it goes away, the problem was in your
> shared system tables. If it persists, the problem is in your backup
> files, and I would try a delete and a fresh copy. If *that* fixes
> it you know the problem was with rdiff-backup. (Of course, keeping
> copies of things before the delete might provide useful forensic
> information.)

Yes, I use createdb to recreate just one database. I doubt backup files
could cause such an error, they are plain SQL files.

Today I got another error, so it seems to get worse:

Warning: pg_dump: WARNING: could not write block 188224 of base/2802415579/2802416218 DETAIL: Multiple failures --- write error might be permanent. pg_dump: SQL command failed pg_dump: Error message from server: ERROR: xlog flush request 200EB/9E4CD48 is not satisfied --- flushed only to CC/3F22EFB4 LINE 1: ...LECT tableoid, oid, nspname, (SELECT rolname FROM pg_catalog... ^ CONTEXT: writing block 188224 of relation base/2802415579/2802416218 pg_dump: The command was: SELECT tableoid, oid, nspname, (SELECT rolname FROM pg_catalog.pg_roles WHERE oid = nspowner) AS rolname, nspacl FROM pg_namespace pg_dumpall: pg_dump failed on database "hekotekerp", exiting
Warning: Failed to dump pgsql cluster

Strange that I have no problems actually using that database.

>
>> For a while development server was running 8.4 and live server
>> 8.1. Now both are 8.4, but this shouldn't matter, as I do backup
>> and restore via SQL.
>
> I hope you were using the 8.4 version of pg_dump when you were in
> the dual-version situation. Using the earlier version of pg_dump is
> not guaranteed to provide a backup which can be cleanly installed on
> a later version. That could *possibly* be related to current
> problems.

I used 8.1 version of pg_dump previously, but had no problems with it.
Currently both are 8.4, so this is not a problem.

>> Development server contains some additional databases as well,
>> that do not exist on live server.
>
> So are you really using pg_dumpall or pg_dump?

I'm using pg_dump on live server and pg_dumpall on development server.

>
>> It's not ECC memory.
>
> Well, then there has been proven to be a non-negligible possibility
> of occasional random bit-flips. Seriously, next time you upgrade,
> make sure any database server has ECC RAM.

Thanks for a tip, will do that.

>
>> It is possible, that restore of pre-live database using psql lasts
>> so long, that backup of the same database using pg_dump is already
>> kicking in.
>
> Hmmm... You might want to do enough logging of the processes to be
> able to confirm or eliminate that possibility. Dumping an
> incompletely-restored database might generate some odd errors.
>

Thanks Kevin for suggestions, investigating further...

Tambet