Re: Replica (v 9.3.2) crashed with "PANIC: WAL contains references to invalid pages"

2014-02-10 12:29 GMT+01:00 Cassiano, Marco <mcassiano(at)manord(dot)com>:

> I resend the mail with gzipped attachment due to mailing list message
> size limits
>
> --------------------------
>
>
>
> Thank you Mat,
>
>
>
> here are the additional infos :
>
>
>
> 1) All of the three servers (main+2 replicas) are virtual on VMware
> Esxi 5.0
>
> 2) Each server is on a different storage and on different vmware
> hosts
>
> 3) The Log of the primary are with no errors
>
> 4) Attached : pg_controldata output, postgres log, and
> /var/log/messages
>
> 5) Fsck on the colume containing the database folders reports no
> error :
>
>
>
> [root(at)pg64prod_rep /]# umount /dev/sdb1
>
> [root(at)pg64prod_rep /]# fsck -n /dev/sdb1
>
> fsck from util-linux-ng 2.17.2
>
> e2fsck 1.41.12 (17-May-2010)
>
> /dev/sdb1 has gone 193 days without being checked, check forced.
>
> Pass 1: Checking inodes, blocks, and sizes
>
> Pass 2: Checking directory structure
>
> Pass 3: Checking directory connectivity
>
> Pass 4: Checking reference counts
>
> Pass 5: Checking group summary information
>
> /dev/sdb1: 7886/13107200 files (8.3% non-contiguous), 38818825/52428119
> blocks
>
>
>
> Marco
>
>
>
> *Da:* desmodemone [mailto:desmodemone(at)gmail(dot)com]
> *Inviato:* lunedì 10 febbraio 2014 10:30
> *A:* Cassiano, Marco
> *Cc:* pgsql-admin(at)postgresql(dot)org
> *Oggetto:* Re: [ADMIN] Replica (v 9.3.2) crashed with "PANIC: WAL
> contains references to invalid pages"
>
>
>
>
>
>
>
> 2014-02-10 9:30 GMT+01:00 Cassiano, Marco <mcassiano(at)manord(dot)com>:
>
>
>
> Hello,
>
> please, could you post some details about? Your replica are on
> the same storage as the primary ? Are virtual or physical ? hypervisor
> type?
>
> Could you attach / post /var/log/messages and postgres log ?
>
> Could you attach / post pg_controldata output of the replica ?
>
> Did you verify the filesystem integrity of the replica ?
>
>
>
> Are the log of the primary without errors ?
>
>
>
> Thank you very much
>
> Mat
>

Hi Marco,

Thanks for your log files about the problem and sorry for late answer, in
the source code of xlogutils.c I see that more log information are logged
at DEBUG2 level of

log_min_messages <= DEBUG2 || client_min_messages <= DEBUG2

So I think will be a good idea to modify your postgresql.conf of one
standby at set the log_min_messages= DEBUG2 , so the code will invoke the
function report_invalid_page and more information
will be reported about the invalid page of the relation on which
investigate.

Bye

Mat