Re: Replica (v 9.3.2) crashed with "PANIC: WAL contains references to invalid pages"

2014-02-10 9:30 GMT+01:00 Cassiano, Marco <mcassiano(at)manord(dot)com>:

> Hello everybody,
>
>
>
> This weeked both replicas of our main db crashed at the same time with
> this error :
>
>
>
> 2014-02-09 11:42:51 GMT 0 52c671da.14da - PANIC: WAL contains
> references to invalid pages
>
> 2014-02-09 11:42:51 GMT 0 52c671da.14da - CONTEXT: xlog redo vacuum:
> rel 1663/16433/29449; blk 181466, lastBlockVacuumed 181463
>
> 2014-02-09 11:42:52 GMT 0 52c671d9.14d1 - LOG: startup process (PID
> 5338) was terminated by signal 6: Aborted
>
> 2014-02-09 11:42:52 GMT 0 52c671d9.14d1 - LOG: terminating any other
> active server processes
>
>
>
>
>
> All three servers (main + two replicas) are on v. 9.3.2 running on Centos
> 6.4
>
>
>
> We upgraded one month ago the main db from v 9.2.6 to 9.3.2 through
> pg_upgrade and had the replicas rebuilt on 9.3.2
>
>
>
> I searched the mailing lists and found someone that had the same problem
> in the past but it seems that their problem was fixed by already released
> patches.
>
>
>
> ( see thread
> http://www.postgresql.org/message-id/675b7cee-b7f0-4e32-8e34-1efaf3ca5fe9@email.android.com
> )
>
>
>
> So it seems that our problem is a new one since we are running the latest
> version.......
>
>
>
> Thank you for your help
>
>
>
>
>
> Marco Cassiano
>
> Manifatture del Nord srl unipersonale
> Gruppo MaxMara
>
> via Mazzacurati 6
>
> C.P. n° 20 - San Maurizio
>
> 42122 Reggio Emilia RE
> ITALY
>
> Tel. +39 0522 358215
> Fax +39 0522 268715
> email : mcassiano(at)manord(dot)com
>
>
>
> [image: cid:image001(dot)jpg(at)01CED0CA(dot)EA27C850] <http://www.pennyblack.com/>
>
>
>
> [image: Max & Co. logo] <http://www.maxandco.com/it/Home>
>
>
>
>
> ---------------------------------------------------------------------------------------------
>
> Il contenuto della presente comunicazione è riservato e destinato
> esclusivamente ai destinatari indicati. Nel caso in cui sia ricevuto da
> persona diversa dal destinatario sono proibite la diffusione, la
> distribuzione e la copia. Nel caso riceveste la presente per errore, Vi
> preghiamo di informarci e di distruggerlo e/o cancellarlo dal Vostro
> computer, senza utilizzare i dati contenuti.
>
> La presente comunicazione (comprensiva dei documenti allegati) non avrà
> valore di proposta contrattuale e/o accettazione di proposte provenienti
> dal destinatario, nè rinuncia o riconoscimento di diritti, debiti e/o
> crediti, nè sarà impegnativa, qualora non sia sottoscritto successivo
> accordo da chi può validamente obbligarci. Non deriverà alcuna
> responsabilità precontrattuale a ns. carico, se la presente non sia seguita
> da contratto sottoscritto dalle parti.
>
>
> ---------------------------------------------------------------------------------------------
>
> The content of the above communication is strictly confidential and
> reserved solely for the referred addressees. In the event of receipt by
> persons different from the addressee, copying, alteration and distribution
> are forbidden. If received by mistake we ask you to inform us and to
> destroy and/or delete from your computer without using the data herein
> contained. The present message (eventual annexes inclusive) shall not be
> considered a contractual proposal and/or acceptance of offer from the
> addressee, nor waiver recognizance of rights, debts and/or credits, nor
> shall it be binding when not executed as a subsequent agreement by persons
> who could lawfully represent us. No pre-contractual liability shall apply
> to us when the present communication is not followed by any binding
> agreement between the parties.
>
>
> ---------------------------------------------------------------------------------------------
>
>
>

Hello,
please, could you post some details about? Your replica are on
the same storage as the primary ? Are virtual or physical ? hypervisor
type?

Could you attach / post /var/log/messages and postgres log ?
Could you attach / post pg_controldata output of the replica ?
Did you verify the filesystem integrity of the replica ?

Are the log of the primary without errors ?

Thank you very much

Mat