Re: CORRUPTION on TOAST table

Hello Adrian, thanks for the response.

master data also located on SAN

Yes, each replica is it own VM with its own virtual disk/volume as served
up from the same SAN

Raw disk mappings are a way for ESX to present a SAN volume directly to a
VM instead of creating a virtual disk.

no unexpected messages detected.

On Sun, Apr 3, 2016 at 11:23 PM, Adrian Klaver <adrian(dot)klaver(at)aklaver(dot)com>
wrote:

> On 04/02/2016 08:38 PM, Soni M wrote:
>
>> Hello Everyone,
>>
>> We face TOAST table corruption.
>>
>> One master and two streaming replicas. The corruption happen only on
>> both streaming replicas.
>>
>> We did found the corrupted rows. Selecting on this row, return (on both
>> replica) : unexpected chunk number 0 (expected 1) for toast value
>> 1100613112 in pg_toast_112517
>> selecting this row on master does not return corruption error, but
>> return correct result instead.
>>
>> Previously, dump on a replica return : unexpected chunk number 0
>> (expected 1) for toast value 3234098599 in pg_toast_112517 (please note
>> the toast value is different)
>>
>> This table size is 343 GB, contain around 206,179,697 live tuples. We
>> found that the corruption happen on the biggest column (this column and
>> its pkey sized around 299 GB total).
>>
>>
> on both replica :
>> fsync NEVER turned off.
>> none unexpected power loss nor OS crash.
>>
>> How can the corruption occurs ? and how can I resolve them ?
>>
>
> Meant to add to previous post.
>
> Do you see anything in the replica Postgres logs that indicate a problem
> with the replication process?
>
> Or any other unexpected messages prior to the point you did the select on
> the replica(s)?
>
>
>
>> Thank so much for the help.
>>
>> Cheers \o/
>>
>> --
>> Regards,
>>
>> Soni Maula Harriz
>>
>
>
> --
> Adrian Klaver
> adrian(dot)klaver(at)aklaver(dot)com
>

--
Regards,

Soni Maula Harriz