Re: pg_rewind after promote

On Thu, 2024-03-28 at 15:52 +0100, Emond Papegaaij wrote:

>  * we detach the primary database backend, forcing a failover
>  * pgpool selects a new primary database and promotes it
>  * the other 2 nodes (the old primary and the other standby) are rewound
> and streaming is resumed from the new primary
>  * the node that needed to be taken out of the cluster (the old primary)
> is shutdown and rebooted
>
> This works fine most of the time, but sometimes we see this message on one of the nodes:
> pg_rewind: source and target cluster are on the same timeline pg_rewind: no rewind required
> This message seems timing related, as the first node might report that,
> while the second reports something like:
> pg_rewind: servers diverged at WAL location 5/F28AB1A8 on timeline 21
> pg_rewind: rewinding from last common checkpoint at 5/F27FCA98 on timeline 21
> pg_rewind: Done!
>
> If we ignore the response from pg_rewind, streaming will break on the node that reported
> no rewind was required. On the new primary, we do observe the database moving from timeline
> 21 to 22, but it seems this takes some time to materialize to be observable by pg_rewind.
>
> 1. Is my observation about the starting of a new timeline correct?
> 2. If yes, is there anything we can do during to block promotion process until the new
> timeline has fully materialized, either by waiting or preferably forcing the new
> timeline to be started?

This must be the problem addressed by commit 009eeee746 [1].

You'd have to upgrade to PostgreSQL v16, which would be a good idea anyway, given
that you are running v12.

A temporary workaround could be to explicitly trigger a checkpoint right after
promotion.

Yours,
Laurenz Albe

[1]. https://git.postgresql.org/gitweb/?p=postgresql.git;a=commit;h=009eeee746825090ec7194321a3db4b298d6571e