Re: Missing Toast Chunk

Sam Nelson <samn(at)consistentstate(dot)com> writes:
> We've got a bit of a problem on a customer's production box. We got a
> "missing chunk number 0 for toast value N" (N being a number) this week on
> their production box. We verified that it was only a problem with one row,
> tried to fix it with updates, and ended up deleting the row.
> ...
> We found the same problem in a couple of other tables, but the big problem
> is that the same table that we just fixed had that error again, in a
> different row this time.

Did you try reindexing that table's toast table? If the problem is a
corrupt index rather than an actually missing toast row, then this would
fix it. Index corruption could also explain multiple such failures in the
same table.

If it becomes clear that there are multiple/continuing occurrences of
corruption, then you've most likely got an unreliable storage system.
It would be a wise idea to go looking for other indicators of
inconsistency and lost rows (dangling foreign-key references for instance).

> Some information on the customer's box: It's an Amazon EC2 box running
> debian (I believe debian 5, but I'm not sure). They are using postgres
> 8.3.11, installed from apt. They are mainly using ruby on rails for their
> application(s).

A number of people around here view EC2 with deep suspicion. It's great
if you only need semi-reliable computing cycles ...

regards, tom lane