CORRUPTION on TOAST table

Hello Everyone,

We face TOAST table corruption.

One master and two streaming replicas. The corruption happen only on both
streaming replicas.

We did found the corrupted rows. Selecting on this row, return (on both
replica) : unexpected chunk number 0 (expected 1) for toast value
1100613112 in pg_toast_112517
selecting this row on master does not return corruption error, but return
correct result instead.

Previously, dump on a replica return : unexpected chunk number 0 (expected
1) for toast value 3234098599 in pg_toast_112517 (please note the toast
value is different)

This table size is 343 GB, contain around 206,179,697 live tuples. We found
that the corruption happen on the biggest column (this column and its pkey
sized around 299 GB total).

replica1 :
ESX 5.5, VM Version 8
Intel(R) Xeon(R) CPU E5649 @ 2.53GHz
8GB RAM
Storage – Raw Disk Mapping in ESX from 3PAR 7400 SAN using Fast Class (10k)
disk
Each volume (single disk as presented by SAN) on the VMs is its own LVM
volume.

replica2 :
ESX 5.5, VM Version 8
Intel(R) Xeon(R) CPU E5-2670 v2 @ 2.50GHz
8GB RAM
Raw Disk Mapping in ESX from 3PAR 7400 SAN using Fast Clkass (10k) disk
Each volume (single disk as presented by SAN) on the VMs is its own LVM
volume.

on both replica :
fsync NEVER turned off.
none unexpected power loss nor OS crash.

How can the corruption occurs ? and how can I resolve them ?

Thank so much for the help.

Cheers \o/

--
Regards,

Soni Maula Harriz