R: Replica (v 9.3.2) crashed with "PANIC: WAL contains references to invalid pages"

I resend the mail with gzipped attachment due to mailing list message size limits
--------------------------

Thank you Mat,

here are the additional infos :

1) All of the three servers (main+2 replicas) are virtual on VMware Esxi 5.0

2) Each server is on a different storage and on different vmware hosts

3) The Log of the primary are with no errors

4) Attached : pg_controldata output, postgres log, and /var/log/messages

5) Fsck on the colume containing the database folders reports no error :

[root(at)pg64prod_rep /]# umount /dev/sdb1
[root(at)pg64prod_rep /]# fsck -n /dev/sdb1
fsck from util-linux-ng 2.17.2
e2fsck 1.41.12 (17-May-2010)
/dev/sdb1 has gone 193 days without being checked, check forced.
Pass 1: Checking inodes, blocks, and sizes
Pass 2: Checking directory structure
Pass 3: Checking directory connectivity
Pass 4: Checking reference counts
Pass 5: Checking group summary information
/dev/sdb1: 7886/13107200 files (8.3% non-contiguous), 38818825/52428119 blocks

Marco

Da: desmodemone [mailto:desmodemone(at)gmail(dot)com]
Inviato: lunedì 10 febbraio 2014 10:30
A: Cassiano, Marco
Cc: pgsql-admin(at)postgresql(dot)org
Oggetto: Re: [ADMIN] Replica (v 9.3.2) crashed with "PANIC: WAL contains references to invalid pages"

2014-02-10 9:30 GMT+01:00 Cassiano, Marco <mcassiano(at)manord(dot)com<mailto:mcassiano(at)manord(dot)com>>:

Hello,
please, could you post some details about? Your replica are on the same storage as the primary ? Are virtual or physical ? hypervisor type?
Could you attach / post /var/log/messages and postgres log ?
Could you attach / post pg_controldata output of the replica ?
Did you verify the filesystem integrity of the replica ?

Are the log of the primary without errors ?

Thank you very much
Mat