Re: CORRUPTION on TOAST table

On 04/02/2016 08:38 PM, Soni M wrote:
> Hello Everyone,
>
> We face TOAST table corruption.
>
> One master and two streaming replicas. The corruption happen only on
> both streaming replicas.
>
> We did found the corrupted rows. Selecting on this row, return (on both
> replica) : unexpected chunk number 0 (expected 1) for toast value
> 1100613112 in pg_toast_112517
> selecting this row on master does not return corruption error, but
> return correct result instead.
>
> Previously, dump on a replica return : unexpected chunk number 0
> (expected 1) for toast value 3234098599 in pg_toast_112517 (please note
> the toast value is different)
>
> This table size is 343 GB, contain around 206,179,697 live tuples. We
> found that the corruption happen on the biggest column (this column and
> its pkey sized around 299 GB total).
>

> on both replica :
> fsync NEVER turned off.
> none unexpected power loss nor OS crash.
>
> How can the corruption occurs ? and how can I resolve them ?

Meant to add to previous post.

Do you see anything in the replica Postgres logs that indicate a problem
with the replication process?

Or any other unexpected messages prior to the point you did the select
on the replica(s)?

>
> Thank so much for the help.
>
> Cheers \o/
>
> --
> Regards,
>
> Soni Maula Harriz

--
Adrian Klaver
adrian(dot)klaver(at)aklaver(dot)com