Prevent pg_rewind destroying the data

From: Christopher Pereira <kripper(at)imatronix(dot)cl>
To: pgsql-general(at)postgresql(dot)org
Subject: Prevent pg_rewind destroying the data
Date: 2020-12-20 19:11:25
Message-ID: f4fb86c6-b219-5cad-5dc3-c9c1a720fdb2@imatronix.cl
Views: Raw Message | Whole Thread | Download mbox | Resend email
Thread:
Lists: pgsql-general

Hi,

When pg_rewind is interrupted due to network errors, the cluster gets
corrupted:

Running pg_rewind for a second time returns "pg_rewind: fatal: target
server must be shut down cleanly".

Trying to fix the cluster with "/usr/pgsql-12/bin/postmaster' --single
-F -D '/var/lib/pgsql/12/mydb' -c archive_mode=on -c
archive_command=false" throws:

LOG:  could not read from log segment 0000003B000000000000003E,
offset 0: read 0 of 8192
LOG:  invalid primary checkpoint record
PANIC:  could not locate a valid checkpoint record

When a cluster failsover because of a network problem, chances are high
that another network problem may occur while we run pg_rewind.
It would be nice if pg_rewind wouldn't destroy the data and leave the
cluster in a state where retrying pg_rewind can succeed.

As a workaround we are thinking in taking a LVM snapshot or do a "cp
--reflink" before running pg_rewind and restore if there is a failure,
but it would be nice if pg_rewind were "non destructive".

Is this possible?
Am I missing something?

We are using PG 12.

Browse pgsql-general by date

  From Date Subject
Next Message Andres Freund 2020-12-20 23:13:08 Re: [PATCH] Logical decoding of TRUNCATE
Previous Message Marcin Giedz 2020-12-20 18:52:01 some questions regarding replication issues and timeline/history files