Re: Streaming replication: PANIC on tertiary when secondary promoted

From: Vijaykumar Jain <vijaykumarjain(dot)github(at)gmail(dot)com>
To: Alexey Bashtanov <bashtanov(at)imap(dot)cc>
Cc: PostgreSQL General <pgsql-general(at)postgresql(dot)org>
Subject: Re: Streaming replication: PANIC on tertiary when secondary promoted
Date: 2021-06-16 16:20:34
Message-ID: CAM+6J95P19h4Xk++X33UyYnsuzNTDU5kn-nUMDao49CBvq63bQ@mail.gmail.com
Views: Raw Message | Whole Thread | Download mbox | Resend email
Thread:
Lists: pgsql-general

What is your recovery_target_timeline set to on replicas ?

I just did a primary -> replica -> cascading replica setup. and then
promoted replica as new primary.
cascading replica was working fine, no restarts required.

for me recovery_target_timeline was set to 'latest'

i have pg14beta installed btw.

initdb -D primary
The files belonging to this database system will be owned by user "postgres".
This user must also own the server process.

postgres(at)db:~/playground$ pg_ctl -D primary -l logfile start
waiting for server to start.... done
server started
postgres(at)db:~/playground$ psql
psql (14beta1)
Type "help" for help.

postgres=# select pg_create_physical_replication_slot('replica');
pg_create_physical_replication_slot
-------------------------------------
(replica,)
(1 row)

postgres=# create table t(id int primary key);
CREATE TABLE
postgres=# insert into t select x from generate_series(1, 100) x; checkpoint;
INSERT 0 100
postgres=# \q

-- create a replica
postgres(at)db:~/playground$ pg_basebackup -D replica -R -X stream -S
replica -v -d "dbname=postgres port=5432" -U postgres
pg_basebackup: initiating base backup, waiting for checkpoint to complete
pg_basebackup: checkpoint completed
pg_basebackup: write-ahead log start point: 0/2000028 on timeline 1
pg_basebackup: starting background WAL receiver
pg_basebackup: write-ahead log end point: 0/2000138
pg_basebackup: waiting for background process to finish streaming ...
pg_basebackup: syncing data to disk ...
pg_basebackup: renaming backup_manifest.tmp to backup_manifest
pg_basebackup: base backup completed
postgres(at)db:~/playground$ vim replica/postgresql.conf

--start the replica (port 5433)
postgres(at)db:~/playground$ pg_ctl -D replica -l replicalog start
waiting for server to start.... done
server started
postgres(at)db:~/playground$ psql -p 5433 -c 'select count(1) from t;'
count
-------
100
(1 row)

postgres(at)db:~/playground$ psql -p 5433
psql (14beta1)
Type "help" for help.

-- create a replica slot for cascading streaming replication
postgres=# select pg_create_physical_replication_slot('cascading_replica');
pg_create_physical_replication_slot
-------------------------------------
(cascading_replica,)
(1 row)

postgres=# \q

-- create a cascading replica off replica

postgres(at)db:~/playground$ pg_basebackup -D cascading_replica -R -X
stream -S cascading_replica -v -d "dbname=postgres port=5433" -U
postgres
pg_basebackup: initiating base backup, waiting for checkpoint to complete
pg_basebackup: checkpoint completed
pg_basebackup: write-ahead log start point: 0/3000028 on timeline 1
pg_basebackup: starting background WAL receiver
pg_basebackup: write-ahead log end point: 0/30000D8
pg_basebackup: waiting for background process to finish streaming ...
pg_basebackup: syncing data to disk ...
pg_basebackup: renaming backup_manifest.tmp to backup_manifest
pg_basebackup: base backup completed
postgres(at)db:~/playground$ vim cascading_replica/postgresql.conf
postgres(at)db:~/playground$ pg_ctl -D cascading_replica -l creplica start
waiting for server to start.... done
server started

-- validate receiving data fine.
postgres(at)db:~/playground$ psql -p 5434 -c 'select count(1) from t;'
count
-------
100
(1 row)

-- stop primary
postgres(at)db:~/playground$ pg_ctl -D primary -l logfile stop
waiting for server to shut down.... done
server stopped

-- promote replica to new primary
postgres(at)db:~/playground$ psql -p 5433
psql (14beta1)
Type "help" for help.

postgres=# select pg_promote();
pg_promote
------------
t
(1 row)

postgres=# select pg_is_in_recovery();
pg_is_in_recovery
-------------------
f
(1 row)

postgres=# \q

--do some dml, validate changes replayed to new replica.
postgres(at)db:~/playground$ psql -p 5434 -c 'select count(1) from t;'
count
-------
100
(1 row)

postgres(at)db:~/playground$ psql -p 5433 -c 'select count(1) from t;'
count
-------
100
(1 row)

postgres(at)db:~/playground$ psql -p 5433 -c 'delete from t where id < 50;'
DELETE 49
postgres(at)db:~/playground$ psql -p 5433 -c 'select count(1) from t;'
count
-------
51
(1 row)

postgres(at)db:~/playground$ psql -p 5434 -c 'select count(1) from t;'
count
-------
51
(1 row)

in all my cases. recovery_timeline was set to 'latest'.
i did not rx any panic messages in logs.

In response to

Responses

Browse pgsql-general by date

  From Date Subject
Next Message Alexey Bashtanov 2021-06-16 19:31:02 Re: Streaming replication: PANIC on tertiary when secondary promoted
Previous Message Alexey Bashtanov 2021-06-16 15:27:35 Streaming replication: PANIC on tertiary when secondary promoted