Re: San replication corrupting postgres file...

On Mon, May 1, 2017 at 1:32 PM, Rahul Sharma <rahulsharma0525(at)gmail(dot)com> wrote:
> Hi Team,
>
> I am facing an issue with postgres replication between my primary and DR
> site. I have the following setup,
>
> 1. I am trying to replicate LVM level sanpshot on SAN which does a block
> level replication.
> 2. OS Details : RHEL 7.1 kernel 3.10
> 3. Postgres Version : ( 9.6)
>
> The steps performed:
>
> 1. Stop all the containers running on the OS.
> 2. Stop the SAN level replication.
> 3. Switch over to the replicated site.
> 4. Start the containers
>
> Here the postgres container fails with the blow error which looks like data
> corruption.
>
> ========
>
> LOG: database system was interrupted; last known up at 2017-04-28 15:58:45
> UTC
> LOG: invalid magic number 7270 in log segment 000000010000000000000001,
> offset 0
> LOG: invalid primary checkpoint record
> LOG: invalid magic number 7270 in log segment 000000010000000000000001,
> offset 0
> LOG: invalid secondary checkpoint record
> PANIC: could not locate a valid checkpoint record
> LOG: startup process (PID 18) was terminated by signal 6: Aborted
> LOG: aborting startup due to startup process failure
> LOG: database system is shut down
>
> =======
>
> I have tried the graceful shutdown of the microservices but still the
> replication fails. Strange issues id i have other instance of postgres
> (9.4.1 )which runs absolutely fine. Could someone please provide some
> advice?

Are your pg xlog and data directories on different volumes? If so then
vm snapshots are likely to not be coherent due to timing etc.

Is there a reason you're NOT using pgsql's built in streaming replication?