<div><div>Unfortunately, the playback is not very stable, but sometimes it shoots. I added some commands to show last WAL rows</div><div>script</div><div>```bash</div><div><div>#!/bin/sh -eux</div><div> </div><div># Goto script's directory</div><div>cd "$(dirname "$0")"</div><div># Stop master if exists</div><div>test -f master/postmaster.pid && pg_ctl -w -D master stop -m fast || echo $?</div><div># Remove master directory</div><div>rm -rf master</div><div># Create master directory</div><div>mkdir -p master</div><div># Initialize master, data checksums are mandatory</div><div>pg_ctl initdb -D master -o "--data-checksums -N -A trust --wal-segsize 1"</div><div># Debug to see recycling WAL, keep only 5 WAL files (because segsize 1MB)</div><div>cat >>master/postgresql.conf <<EOF</div><div>hot_standby = on</div><div>logging_collector = on</div><div>log_min_messages = debug</div><div>#synchronous_standby_names = postgres</div><div>wal_keep_size = 5MB</div><div>EOF</div><div># Accept replication connections on master</div><div>cat >> master/pg_hba.conf <<EOF</div><div>local replication all trust</div><div>host replication all 127.0.0.1/32 trust</div><div>host replication all ::1/128 trust</div><div>EOF</div><div># Start master</div><div>pg_ctl -w -D master start -o "-p 5000"</div><div># Stop standby1 if exists</div><div>test -f standby1/postmaster.pid && pg_ctl -w -D standby1 stop -m fast || echo $?</div><div># Remove standby1 directory</div><div>rm -rf standby1</div><div># Base backup is taken with xlog files included</div><div>pg_basebackup -D standby1 -p 5000 -X fetch --verbose --write-recovery-conf</div><div># Start standby1</div><div>pg_ctl -w -D standby1 start -o "-p 5001"</div><div># Stop standby2 if exists</div><div>test -f standby2/postmaster.pid && pg_ctl -w -D standby2 stop -m fast || echo $?</div><div># Remove standby2 directory</div><div>rm -rf standby2</div><div># Base backup is taken with xlog files included</div><div>pg_basebackup -D standby2 -p 5000 -X fetch --verbose --write-recovery-conf</div><div># Start standby2</div><div>pg_ctl -w -D standby2 start -o "-p 5002"</div><div># Create table and insert lot of rows</div><div>psql -d postgres -a --no-psqlrc -p 5000 <<EOF</div><div>create table a (a int, b int);</div><div>insert into a select i, i from generate_series(1, 100000) i;</div><div>EOF</div><div># Stop master to recycle WAL files</div><div>pg_ctl -w -D master stop -m fast</div><div># Stop standby1 to recycle WAL files</div><div>pg_ctl -w -D standby1 stop -m fast</div><div># Stop standby2 to recycle WAL files</div><div>pg_ctl -w -D standby2 stop -m fast</div><div># Start master</div><div>pg_ctl -w -D master start -o "-p 5000"</div><div># Start standby1</div><div>pg_ctl -w -D standby1 start -o "-p 5001"</div><div># Start standby2</div><div>pg_ctl -w -D standby2 start -o "-p 5002"</div><div># Find latest master WAL file</div><div>LAST_MASTER_WAL_FILE="$(pg_controldata master | grep "Latest checkpoint's REDO WAL file:" | cut -d : -f 2 | grep -oP "\w+")"</div><div># Now promote standby1</div><div>pg_ctl -w -D standby1 promote</div><div># Wait until standby1 is promoted</div><div>while ! pg_isready -p 5001; do sleep 1; done</div><div># Find latest standby1 WAL file</div><div>LAST_STANDBY_WAL_FILE="$(pg_controldata standby1 | grep "Latest checkpoint's REDO WAL file:" | cut -d : -f 2 | grep -oP "\w+")"</div><div># Print last timeline history</div><div>cat "$(find standby1/pg_wal -name "*.history" -type f -print0 | xargs -r -0 ls -1 -t | head -1)"</div><div># Print control information of master</div><div>pg_controldata master | grep "checkpoint" | grep "location"</div><div># Print control information of standby1</div><div>pg_controldata standby1 | grep "checkpoint" | grep "location"</div><div># Print control information of standby2</div><div>pg_controldata standby2 | grep "checkpoint" | grep "location"</div><div># Print last 10 lines from dumped master WAL file</div><div>pg_waldump -p master "$LAST_MASTER_WAL_FILE" | tail -n 10</div><div># Print last 10 lines from dumped standby1 WAL files</div><div>pg_waldump -p standby1 "$LAST_MASTER_WAL_FILE" | tail -n 10</div><div>pg_waldump -p standby1 "$LAST_STANDBY_WAL_FILE" | tail -n 10</div><div># Print last 10 lines from dumped standby2 WAL file</div><div>pg_waldump -p standby2 "$LAST_MASTER_WAL_FILE" | tail -n 10</div><div># Insert some rows</div><div>psql -d postgres -a --no-psqlrc -p 5000 <<EOF</div><div>--set synchronous_standby_names = 'postgres';</div><div>--select pg_reload_conf();</div><div>insert into a select i, i from generate_series(1, 340) i;</div><div>--commit;</div><div>EOF</div><div># Standby2 node need to be stopped as well.</div><div>pg_ctl -w -D standby2 stop -m fast</div><div># Stop the master after standby promotion</div><div>pg_ctl -w -D master stop -m fast</div><div># Standby1 node need to be stopped as well.</div><div>pg_ctl -w -D standby1 stop -m fast</div><div># Print control information of master</div><div>pg_controldata master | grep "checkpoint" | grep "location"</div><div># Print control information of standby1</div><div>pg_controldata standby1 | grep "checkpoint" | grep "location"</div><div># Print control information of standby2</div><div>pg_controldata standby2 | grep "checkpoint" | grep "location"</div><div># Print last 10 lines from dumped master WAL file</div><div>pg_waldump -p master "$LAST_MASTER_WAL_FILE" | tail -n 10</div><div># Print last 10 lines from dumped standby1 WAL files</div><div>pg_waldump -p standby1 "$LAST_MASTER_WAL_FILE" | tail -n 10</div><div>pg_waldump -p standby1 "$LAST_STANDBY_WAL_FILE" | tail -n 10</div><div># Print last 10 lines from dumped standby2 WAL file</div><div>pg_waldump -p standby2 "$LAST_MASTER_WAL_FILE" | tail -n 10</div><div># Do rewind using standby1 pgdata as source</div><div>pg_rewind --progress --debug --source-pgdata=standby1 --target-pgdata=standby2</div><div># Parameters for standby2 postgresql.conf</div><div>cat >> standby2/postgresql.conf <<EOF</div><div>primary_conninfo = 'port=5001'</div><div>EOF</div><div># Place standby2 signal file</div><div>touch standby2/standby.signal</div><div># Start standby1</div><div>pg_ctl -w -D standby1 start -o "-p 5001"</div><div># Start standby2</div><div>pg_ctl -w -D standby2 start -o "-p 5002"</div><div># Same query</div><div>psql -d postgres -a --no-psqlrc -p 5001 <<EOF</div><div>select count(*) from a;</div><div>EOF</div><div># Different results</div><div>psql -d postgres -a --no-psqlrc -p 5002 <<EOF</div><div>select count(*) from a;</div><div>EOF</div><div># Stop standby1</div><div>pg_ctl -w -D standby1 stop -m fast</div><div># Stop standby2</div><div>pg_ctl -w -D standby2 stop -m fast</div></div><div>```</div><div>output</div><div>```bash</div><div><div>+ dirname /var/lib/postgresql/ADBDEV/5716.1/r16_2.sh</div><div>+ cd /var/lib/postgresql/ADBDEV/5716.1</div><div>+ test -f master/postmaster.pid</div><div>+ echo 1</div><div>1</div><div>+ rm -rf master</div><div>+ mkdir -p master</div><div>+ pg_ctl initdb -D master -o '--data-checksums -N -A trust --wal-segsize 1'</div><div>The files belonging to this database system will be owned by user "postgres".</div><div>This user must also own the server process.</div><div> </div><div>The database cluster will be initialized with locale "ru_RU.UTF-8".</div><div>The default database encoding has accordingly been set to "UTF8".</div><div>The default text search configuration will be set to "russian".</div><div> </div><div>Data page checksums are enabled.</div><div> </div><div>fixing permissions on existing directory master ... ok</div><div>creating subdirectories ... ok</div><div>selecting dynamic shared memory implementation ... posix</div><div>selecting default max_connections ... 100</div><div>selecting default shared_buffers ... 128MB</div><div>selecting default time zone ... Asia/Yekaterinburg</div><div>creating configuration files ... ok</div><div>running bootstrap script ... ok</div><div>performing post-bootstrap initialization ... ok</div><div> </div><div>Sync to disk skipped.</div><div>The data directory might become corrupt if the operating system crashes.</div><div> </div><div>Success. You can now start the database server using:</div><div> </div><div> /usr/local/bin/pg_ctl -D master -l logfile start</div><div> </div><div>+ cat</div><div>+ cat</div><div>+ pg_ctl -w -D master start -o '-p 5000'</div><div>waiting for server to start....2024-08-08 12:52:03.032 +05 [1911] DEBUG: registering background worker "logical replication launcher"</div><div>2024-08-08 12:52:03.032 +05 [1911] DEBUG: mmap(146800640) with MAP_HUGETLB failed, huge pages disabled: Out of memory</div><div>2024-08-08 12:52:03.035 +05 [1911] DEBUG: dynamic shared memory system will support 674 segments</div><div>2024-08-08 12:52:03.035 +05 [1911] DEBUG: created dynamic shared memory control segment 1884847300 (26976 bytes)</div><div>2024-08-08 12:52:03.035 +05 [1911] DEBUG: max_safe_fds = 986, usable_fds = 1000, already_open = 4</div><div>2024-08-08 12:52:03.035 +05 [1911] LOG: redirecting log output to logging collector process</div><div>2024-08-08 12:52:03.035 +05 [1911] HINT: Future log output will appear in directory "log".</div><div> done</div><div>server started</div><div>+ test -f standby1/postmaster.pid</div><div>+ echo 1</div><div>1</div><div>+ rm -rf standby1</div><div>+ pg_basebackup -D standby1 -p 5000 -X fetch --verbose --write-recovery-conf</div><div>pg_basebackup: initiating base backup, waiting for checkpoint to complete</div><div>pg_basebackup: checkpoint completed</div><div>pg_basebackup: write-ahead log start point: 0/800028 on timeline 1</div><div>pg_basebackup: write-ahead log end point: 0/800100</div><div>pg_basebackup: syncing data to disk ...</div><div>pg_basebackup: renaming backup_manifest.tmp to backup_manifest</div><div>pg_basebackup: base backup completed</div><div>+ pg_ctl -w -D standby1 start -o '-p 5001'</div><div>waiting for server to start....2024-08-08 12:52:03.886 +05 [1924] DEBUG: registering background worker "logical replication launcher"</div><div>2024-08-08 12:52:03.886 +05 [1924] DEBUG: mmap(146800640) with MAP_HUGETLB failed, huge pages disabled: Out of memory</div><div>2024-08-08 12:52:03.888 +05 [1924] DEBUG: dynamic shared memory system will support 674 segments</div><div>2024-08-08 12:52:03.889 +05 [1924] DEBUG: created dynamic shared memory control segment 689671304 (26976 bytes)</div><div>2024-08-08 12:52:03.889 +05 [1924] DEBUG: max_safe_fds = 986, usable_fds = 1000, already_open = 4</div><div>2024-08-08 12:52:03.889 +05 [1924] LOG: redirecting log output to logging collector process</div><div>2024-08-08 12:52:03.889 +05 [1924] HINT: Future log output will appear in directory "log".</div><div> done</div><div>server started</div><div>+ test -f standby2/postmaster.pid</div><div>+ echo 1</div><div>1</div><div>+ rm -rf standby2</div><div>+ pg_basebackup -D standby2 -p 5000 -X fetch --verbose --write-recovery-conf</div><div>pg_basebackup: initiating base backup, waiting for checkpoint to complete</div><div>pg_basebackup: checkpoint completed</div><div>pg_basebackup: write-ahead log start point: 0/900028 on timeline 1</div><div>pg_basebackup: write-ahead log end point: 0/900100</div><div>pg_basebackup: syncing data to disk ...</div><div>pg_basebackup: renaming backup_manifest.tmp to backup_manifest</div><div>pg_basebackup: base backup completed</div><div>+ pg_ctl -w -D standby2 start -o '-p 5002'</div><div>waiting for server to start....2024-08-08 12:52:04.870 +05 [1936] DEBUG: registering background worker "logical replication launcher"</div><div>2024-08-08 12:52:04.870 +05 [1936] DEBUG: mmap(146800640) with MAP_HUGETLB failed, huge pages disabled: Out of memory</div><div>2024-08-08 12:52:04.873 +05 [1936] DEBUG: dynamic shared memory system will support 674 segments</div><div>2024-08-08 12:52:04.873 +05 [1936] DEBUG: created dynamic shared memory control segment 111286480 (26976 bytes)</div><div>2024-08-08 12:52:04.873 +05 [1936] DEBUG: max_safe_fds = 986, usable_fds = 1000, already_open = 4</div><div>2024-08-08 12:52:04.874 +05 [1936] LOG: redirecting log output to logging collector process</div><div>2024-08-08 12:52:04.874 +05 [1936] HINT: Future log output will appear in directory "log".</div><div> done</div><div>server started</div><div>+ psql -d postgres -a --no-psqlrc -p 5000</div><div>create table a (a int, b int);</div><div>CREATE TABLE</div><div>insert into a select i, i from generate_series(1, 100000) i;</div><div>INSERT 0 100000</div><div>+ pg_ctl -w -D master stop -m fast</div><div>waiting for server to shut down....2024-08-08 12:52:05.333 +05 [1912] DEBUG: logger shutting down</div><div> done</div><div>server stopped</div><div>+ pg_ctl -w -D standby1 stop -m fast</div><div>waiting for server to shut down....2024-08-08 12:52:05.483 +05 [1925] DEBUG: logger shutting down</div><div> done</div><div>server stopped</div><div>+ pg_ctl -w -D standby2 stop -m fast</div><div>waiting for server to shut down....2024-08-08 12:52:05.585 +05 [1937] DEBUG: logger shutting down</div><div> done</div><div>server stopped</div><div>+ pg_ctl -w -D master start -o '-p 5000'</div><div>waiting for server to start....2024-08-08 12:52:05.628 +05 [1952] DEBUG: registering background worker "logical replication launcher"</div><div>2024-08-08 12:52:05.628 +05 [1952] DEBUG: mmap(146800640) with MAP_HUGETLB failed, huge pages disabled: Out of memory</div><div>2024-08-08 12:52:05.631 +05 [1952] DEBUG: dynamic shared memory system will support 674 segments</div><div>2024-08-08 12:52:05.631 +05 [1952] DEBUG: created dynamic shared memory control segment 1418826688 (26976 bytes)</div><div>2024-08-08 12:52:05.631 +05 [1952] DEBUG: max_safe_fds = 986, usable_fds = 1000, already_open = 4</div><div>2024-08-08 12:52:05.632 +05 [1952] LOG: redirecting log output to logging collector process</div><div>2024-08-08 12:52:05.632 +05 [1952] HINT: Future log output will appear in directory "log".</div><div> done</div><div>server started</div><div>+ pg_ctl -w -D standby1 start -o '-p 5001'</div><div>waiting for server to start....2024-08-08 12:52:05.736 +05 [1962] DEBUG: registering background worker "logical replication launcher"</div><div>2024-08-08 12:52:05.736 +05 [1962] DEBUG: mmap(146800640) with MAP_HUGETLB failed, huge pages disabled: Out of memory</div><div>2024-08-08 12:52:05.739 +05 [1962] DEBUG: dynamic shared memory system will support 674 segments</div><div>2024-08-08 12:52:05.739 +05 [1962] DEBUG: created dynamic shared memory control segment 1087487228 (26976 bytes)</div><div>2024-08-08 12:52:05.739 +05 [1962] DEBUG: max_safe_fds = 986, usable_fds = 1000, already_open = 4</div><div>2024-08-08 12:52:05.739 +05 [1962] LOG: redirecting log output to logging collector process</div><div>2024-08-08 12:52:05.739 +05 [1962] HINT: Future log output will appear in directory "log".</div><div> done</div><div>server started</div><div>+ pg_ctl -w -D standby2 start -o '-p 5002'</div><div>waiting for server to start....2024-08-08 12:52:05.841 +05 [1971] DEBUG: registering background worker "logical replication launcher"</div><div>2024-08-08 12:52:05.841 +05 [1971] DEBUG: mmap(146800640) with MAP_HUGETLB failed, huge pages disabled: Out of memory</div><div>2024-08-08 12:52:05.844 +05 [1971] DEBUG: dynamic shared memory system will support 674 segments</div><div>2024-08-08 12:52:05.844 +05 [1971] DEBUG: created dynamic shared memory control segment 3045047604 (26976 bytes)</div><div>2024-08-08 12:52:05.845 +05 [1971] DEBUG: max_safe_fds = 986, usable_fds = 1000, already_open = 4</div><div>2024-08-08 12:52:05.845 +05 [1971] LOG: redirecting log output to logging collector process</div><div>2024-08-08 12:52:05.845 +05 [1971] HINT: Future log output will appear in directory "log".</div><div> done</div><div>server started</div><div>+ pg_controldata master</div><div>+ grep 'Latest checkpoint'"'"'s REDO WAL file:'</div><div>+ cut -d : -f+ 2grep</div><div> -oP '\w+'</div><div>+ LAST_MASTER_WAL_FILE=000000010000000000000010</div><div>+ pg_ctl -w -D standby1 promote</div><div>waiting for server to promote.... done</div><div>server promoted</div><div>+ pg_isready -p 5001</div><div>/tmp:5001 - accepting connections</div><div>+ pg_controldata standby1</div><div>+ grep 'Latest checkpoint'"'"'s REDO WAL file:'</div><div>+ cut -d : -f 2</div><div>+ grep -oP '\w+'</div><div>+ LAST_STANDBY_WAL_FILE=000000020000000000000010</div><div>+ find standby1/pg_wal -name '*.history' -type f -print0</div><div>+ xargs -r -0 ls -1 -t</div><div>+ head -1</div><div>+ cat standby1/pg_wal/00000002.history</div><div>1 0/10392E8 no recovery target specified</div><div>+ pg_controldata master</div><div>+ grep checkpoint</div><div>+ grep location</div><div>Latest checkpoint location: 0/1039270</div><div>Latest checkpoint's REDO location: 0/1039270</div><div>+ pg_controldata standby1</div><div>+ grep checkpoint</div><div>+ grep location</div><div>Latest checkpoint location: 0/1039350</div><div>Latest checkpoint's REDO location: 0/1039318</div><div>+ pg_controldata standby2</div><div>+ grep checkpoint</div><div>+ grep location</div><div>Latest checkpoint location: 0/1039270</div><div>Latest checkpoint's REDO location: 0/1039270</div><div>+ pg_waldump -p master 000000010000000000000010</div><div>+ tail -n 10</div><div>pg_waldump: error: error in WAL record at 0/1039270: invalid record length at 0/10392E8: expected at least 24, got 0</div><div>rmgr: Heap len (rec/tot): 63/ 63, tx: 731, lsn: 0/01039048, prev 0/01039008, desc: INSERT off: 101, flags: 0x00, blkref #0: rel 1663/5/16384 blk 442</div><div>rmgr: Heap len (rec/tot): 63/ 63, tx: 731, lsn: 0/01039088, prev 0/01039048, desc: INSERT off: 102, flags: 0x00, blkref #0: rel 1663/5/16384 blk 442</div><div>rmgr: Heap len (rec/tot): 63/ 63, tx: 731, lsn: 0/010390C8, prev 0/01039088, desc: INSERT off: 103, flags: 0x00, blkref #0: rel 1663/5/16384 blk 442</div><div>rmgr: Heap len (rec/tot): 63/ 63, tx: 731, lsn: 0/01039108, prev 0/010390C8, desc: INSERT off: 104, flags: 0x00, blkref #0: rel 1663/5/16384 blk 442</div><div>rmgr: Heap len (rec/tot): 63/ 63, tx: 731, lsn: 0/01039148, prev 0/01039108, desc: INSERT off: 105, flags: 0x00, blkref #0: rel 1663/5/16384 blk 442</div><div>rmgr: Heap len (rec/tot): 63/ 63, tx: 731, lsn: 0/01039188, prev 0/01039148, desc: INSERT off: 106, flags: 0x00, blkref #0: rel 1663/5/16384 blk 442</div><div>rmgr: Heap len (rec/tot): 63/ 63, tx: 731, lsn: 0/010391C8, prev 0/01039188, desc: INSERT off: 107, flags: 0x00, blkref #0: rel 1663/5/16384 blk 442</div><div>rmgr: Heap len (rec/tot): 63/ 63, tx: 731, lsn: 0/01039208, prev 0/010391C8, desc: INSERT off: 108, flags: 0x00, blkref #0: rel 1663/5/16384 blk 442</div><div>rmgr: Transaction len (rec/tot): 34/ 34, tx: 731, lsn: 0/01039248, prev 0/01039208, desc: COMMIT 2024-08-08 12:52:05.194389 +05</div><div>rmgr: XLOG len (rec/tot): 114/ 114, tx: 0, lsn: 0/01039270, prev 0/01039248, desc: CHECKPOINT_SHUTDOWN redo 0/1039270; tli 1; prev tli 1; fpw true; xid 0:732; oid 16387; multi 1; offset 0; oldest xid 722 in DB 1; oldest multi 1 in DB 1; oldest/newest commit timestamp xid: 0/0; oldest running xid 0; shutdown</div><div>+ pg_waldump -p standby1 000000010000000000000010</div><div>+ tail -n 10</div><div>pg_waldump: error: error in WAL record at 0/1039270: invalid record length at 0/10392E8: expected at least 24, got 0</div><div>rmgr: Heap len (rec/tot): 63/ 63, tx: 731, lsn: 0/01039048, prev 0/01039008, desc: INSERT off: 101, flags: 0x00, blkref #0: rel 1663/5/16384 blk 442</div><div>rmgr: Heap len (rec/tot): 63/ 63, tx: 731, lsn: 0/01039088, prev 0/01039048, desc: INSERT off: 102, flags: 0x00, blkref #0: rel 1663/5/16384 blk 442</div><div>rmgr: Heap len (rec/tot): 63/ 63, tx: 731, lsn: 0/010390C8, prev 0/01039088, desc: INSERT off: 103, flags: 0x00, blkref #0: rel 1663/5/16384 blk 442</div><div>rmgr: Heap len (rec/tot): 63/ 63, tx: 731, lsn: 0/01039108, prev 0/010390C8, desc: INSERT off: 104, flags: 0x00, blkref #0: rel 1663/5/16384 blk 442</div><div>rmgr: Heap len (rec/tot): 63/ 63, tx: 731, lsn: 0/01039148, prev 0/01039108, desc: INSERT off: 105, flags: 0x00, blkref #0: rel 1663/5/16384 blk 442</div><div>rmgr: Heap len (rec/tot): 63/ 63, tx: 731, lsn: 0/01039188, prev 0/01039148, desc: INSERT off: 106, flags: 0x00, blkref #0: rel 1663/5/16384 blk 442</div><div>rmgr: Heap len (rec/tot): 63/ 63, tx: 731, lsn: 0/010391C8, prev 0/01039188, desc: INSERT off: 107, flags: 0x00, blkref #0: rel 1663/5/16384 blk 442</div><div>rmgr: Heap len (rec/tot): 63/ 63, tx: 731, lsn: 0/01039208, prev 0/010391C8, desc: INSERT off: 108, flags: 0x00, blkref #0: rel 1663/5/16384 blk 442</div><div>rmgr: Transaction len (rec/tot): 34/ 34, tx: 731, lsn: 0/01039248, prev 0/01039208, desc: COMMIT 2024-08-08 12:52:05.194389 +05</div><div>rmgr: XLOG len (rec/tot): 114/ 114, tx: 0, lsn: 0/01039270, prev 0/01039248, desc: CHECKPOINT_SHUTDOWN redo 0/1039270; tli 1; prev tli 1; fpw true; xid 0:732; oid 16387; multi 1; offset 0; oldest xid 722 in DB 1; oldest multi 1 in DB 1; oldest/newest commit timestamp xid: 0/0; oldest running xid 0; shutdown</div><div>+ pg_waldump -p standby1 000000020000000000000010</div><div>+ tail -n 10</div><div>pg_waldump: error: error in WAL record at 0/10393C8: unexpected pageaddr 0/1038000 in WAL segment 000000020000000000000010, LSN 0/103A000, offset 237568</div><div>rmgr: Heap len (rec/tot): 63/ 63, tx: 731, lsn: 0/01039148, prev 0/01039108, desc: INSERT off: 105, flags: 0x00, blkref #0: rel 1663/5/16384 blk 442</div><div>rmgr: Heap len (rec/tot): 63/ 63, tx: 731, lsn: 0/01039188, prev 0/01039148, desc: INSERT off: 106, flags: 0x00, blkref #0: rel 1663/5/16384 blk 442</div><div>rmgr: Heap len (rec/tot): 63/ 63, tx: 731, lsn: 0/010391C8, prev 0/01039188, desc: INSERT off: 107, flags: 0x00, blkref #0: rel 1663/5/16384 blk 442</div><div>rmgr: Heap len (rec/tot): 63/ 63, tx: 731, lsn: 0/01039208, prev 0/010391C8, desc: INSERT off: 108, flags: 0x00, blkref #0: rel 1663/5/16384 blk 442</div><div>rmgr: Transaction len (rec/tot): 34/ 34, tx: 731, lsn: 0/01039248, prev 0/01039208, desc: COMMIT 2024-08-08 12:52:05.194389 +05</div><div>rmgr: XLOG len (rec/tot): 114/ 114, tx: 0, lsn: 0/01039270, prev 0/01039248, desc: CHECKPOINT_SHUTDOWN redo 0/1039270; tli 1; prev tli 1; fpw true; xid 0:732; oid 16387; multi 1; offset 0; oldest xid 722 in DB 1; oldest multi 1 in DB 1; oldest/newest commit timestamp xid: 0/0; oldest running xid 0; shutdown</div><div>rmgr: XLOG len (rec/tot): 42/ 42, tx: 0, lsn: 0/010392E8, prev 0/01039270, desc: END_OF_RECOVERY tli 2; prev tli 1; time 2024-08-08 12:52:05.946660 +05</div><div>rmgr: Standby len (rec/tot): 50/ 50, tx: 0, lsn: 0/01039318, prev 0/010392E8, desc: RUNNING_XACTS nextXid 732 latestCompletedXid 731 oldestRunningXid 732</div><div>rmgr: XLOG len (rec/tot): 114/ 114, tx: 0, lsn: 0/01039350, prev 0/01039318, desc: CHECKPOINT_ONLINE redo 0/1039318; tli 2; prev tli 2; fpw true; xid 0:732; oid 16387; multi 1; offset 0; oldest xid 722 in DB 1; oldest multi 1 in DB 1; oldest/newest commit timestamp xid: 0/0; oldest running xid 732; online</div><div>rmgr: XLOG len (rec/tot): 49/ 885, tx: 0, lsn: 0/010393C8, prev 0/01039350, desc: FPI_FOR_HINT , blkref #0: rel 1663/5/1259 blk 0 FPW</div><div>+ pg_waldump -p standby2 000000010000000000000010</div><div>+ tail -n 10</div><div>pg_waldump: error: error in WAL record at 0/1039270: invalid record length at 0/10392E8: expected at least 24, got 0</div><div>rmgr: Heap len (rec/tot): 63/ 63, tx: 731, lsn: 0/01039048, prev 0/01039008, desc: INSERT off: 101, flags: 0x00, blkref #0: rel 1663/5/16384 blk 442</div><div>rmgr: Heap len (rec/tot): 63/ 63, tx: 731, lsn: 0/01039088, prev 0/01039048, desc: INSERT off: 102, flags: 0x00, blkref #0: rel 1663/5/16384 blk 442</div><div>rmgr: Heap len (rec/tot): 63/ 63, tx: 731, lsn: 0/010390C8, prev 0/01039088, desc: INSERT off: 103, flags: 0x00, blkref #0: rel 1663/5/16384 blk 442</div><div>rmgr: Heap len (rec/tot): 63/ 63, tx: 731, lsn: 0/01039108, prev 0/010390C8, desc: INSERT off: 104, flags: 0x00, blkref #0: rel 1663/5/16384 blk 442</div><div>rmgr: Heap len (rec/tot): 63/ 63, tx: 731, lsn: 0/01039148, prev 0/01039108, desc: INSERT off: 105, flags: 0x00, blkref #0: rel 1663/5/16384 blk 442</div><div>rmgr: Heap len (rec/tot): 63/ 63, tx: 731, lsn: 0/01039188, prev 0/01039148, desc: INSERT off: 106, flags: 0x00, blkref #0: rel 1663/5/16384 blk 442</div><div>rmgr: Heap len (rec/tot): 63/ 63, tx: 731, lsn: 0/010391C8, prev 0/01039188, desc: INSERT off: 107, flags: 0x00, blkref #0: rel 1663/5/16384 blk 442</div><div>rmgr: Heap len (rec/tot): 63/ 63, tx: 731, lsn: 0/01039208, prev 0/010391C8, desc: INSERT off: 108, flags: 0x00, blkref #0: rel 1663/5/16384 blk 442</div><div>rmgr: Transaction len (rec/tot): 34/ 34, tx: 731, lsn: 0/01039248, prev 0/01039208, desc: COMMIT 2024-08-08 12:52:05.194389 +05</div><div>rmgr: XLOG len (rec/tot): 114/ 114, tx: 0, lsn: 0/01039270, prev 0/01039248, desc: CHECKPOINT_SHUTDOWN redo 0/1039270; tli 1; prev tli 1; fpw true; xid 0:732; oid 16387; multi 1; offset 0; oldest xid 722 in DB 1; oldest multi 1 in DB 1; oldest/newest commit timestamp xid: 0/0; oldest running xid 0; shutdown</div><div>+ psql -d postgres -a --no-psqlrc -p 5000</div><div>--set synchronous_standby_names = 'postgres';</div><div>--select pg_reload_conf();</div><div>insert into a select i, i from generate_series(1, 340) i;</div><div>INSERT 0 340</div><div>--commit;</div><div>+ pg_ctl -w -D standby2 stop -m fast</div><div>waiting for server to shut down....2024-08-08 12:52:06.096 +05 [1972] DEBUG: logger shutting down</div><div> done</div><div>server stopped</div><div>+ pg_ctl -w -D master stop -m fast</div><div>waiting for server to shut down....2024-08-08 12:52:06.203 +05 [1953] DEBUG: logger shutting down</div><div> done</div><div>server stopped</div><div>+ pg_ctl -w -D standby1 stop -m fast</div><div>waiting for server to shut down....2024-08-08 12:52:06.302 +05 [1963] DEBUG: logger shutting down</div><div> done</div><div>server stopped</div><div>+ pg_controldata master</div><div>+ grep checkpoint</div><div>+ grep location</div><div>Latest checkpoint location: 0/1041370</div><div>Latest checkpoint's REDO location: 0/1041370</div><div>+ pg_controldata standby1</div><div>+ grep checkpoint</div><div>+ grep location</div><div>Latest checkpoint location: 0/103B2F0</div><div>Latest checkpoint's REDO location: 0/103B2F0</div><div>+ pg_controldata standby2</div><div>+ grep checkpoint</div><div>+ grep location</div><div>Latest checkpoint location: 0/1039270</div><div>Latest checkpoint's REDO location: 0/1039270</div><div>+ pg_waldump -p master 000000010000000000000010</div><div>+ tail -n 10</div><div>pg_waldump: error: error in WAL record at 0/1041370: invalid record length at 0/10413E8: expected at least 24, got 0</div><div>rmgr: Heap len (rec/tot): 63/ 63, tx: 732, lsn: 0/01041148, prev 0/01041108, desc: INSERT off: 215, flags: 0x00, blkref #0: rel 1663/5/16384 blk 443</div><div>rmgr: Heap len (rec/tot): 63/ 63, tx: 732, lsn: 0/01041188, prev 0/01041148, desc: INSERT off: 216, flags: 0x00, blkref #0: rel 1663/5/16384 blk 443</div><div>rmgr: Heap len (rec/tot): 63/ 63, tx: 732, lsn: 0/010411C8, prev 0/01041188, desc: INSERT off: 217, flags: 0x00, blkref #0: rel 1663/5/16384 blk 443</div><div>rmgr: Heap len (rec/tot): 63/ 63, tx: 732, lsn: 0/01041208, prev 0/010411C8, desc: INSERT off: 218, flags: 0x00, blkref #0: rel 1663/5/16384 blk 443</div><div>rmgr: Heap len (rec/tot): 63/ 63, tx: 732, lsn: 0/01041248, prev 0/01041208, desc: INSERT off: 219, flags: 0x00, blkref #0: rel 1663/5/16384 blk 443</div><div>rmgr: Heap len (rec/tot): 63/ 63, tx: 732, lsn: 0/01041288, prev 0/01041248, desc: INSERT off: 220, flags: 0x00, blkref #0: rel 1663/5/16384 blk 443</div><div>rmgr: Heap len (rec/tot): 63/ 63, tx: 732, lsn: 0/010412C8, prev 0/01041288, desc: INSERT off: 221, flags: 0x00, blkref #0: rel 1663/5/16384 blk 443</div><div>rmgr: Heap len (rec/tot): 63/ 63, tx: 732, lsn: 0/01041308, prev 0/010412C8, desc: INSERT off: 222, flags: 0x00, blkref #0: rel 1663/5/16384 blk 443</div><div>rmgr: Transaction len (rec/tot): 34/ 34, tx: 732, lsn: 0/01041348, prev 0/01041308, desc: COMMIT 2024-08-08 12:52:06.085591 +05</div><div>rmgr: XLOG len (rec/tot): 114/ 114, tx: 0, lsn: 0/01041370, prev 0/01041348, desc: CHECKPOINT_SHUTDOWN redo 0/1041370; tli 1; prev tli 1; fpw true; xid 0:733; oid 16387; multi 1; offset 0; oldest xid 722 in DB 1; oldest multi 1 in DB 1; oldest/newest commit timestamp xid: 0/0; oldest running xid 0; shutdown</div><div>+ pg_waldump -p standby1 000000010000000000000010</div><div>+ tail -n 10</div><div>pg_waldump: error: error in WAL record at 0/1039270: invalid record length at 0/10392E8: expected at least 24, got 0</div><div>rmgr: Heap len (rec/tot): 63/ 63, tx: 731, lsn: 0/01039048, prev 0/01039008, desc: INSERT off: 101, flags: 0x00, blkref #0: rel 1663/5/16384 blk 442</div><div>rmgr: Heap len (rec/tot): 63/ 63, tx: 731, lsn: 0/01039088, prev 0/01039048, desc: INSERT off: 102, flags: 0x00, blkref #0: rel 1663/5/16384 blk 442</div><div>rmgr: Heap len (rec/tot): 63/ 63, tx: 731, lsn: 0/010390C8, prev 0/01039088, desc: INSERT off: 103, flags: 0x00, blkref #0: rel 1663/5/16384 blk 442</div><div>rmgr: Heap len (rec/tot): 63/ 63, tx: 731, lsn: 0/01039108, prev 0/010390C8, desc: INSERT off: 104, flags: 0x00, blkref #0: rel 1663/5/16384 blk 442</div><div>rmgr: Heap len (rec/tot): 63/ 63, tx: 731, lsn: 0/01039148, prev 0/01039108, desc: INSERT off: 105, flags: 0x00, blkref #0: rel 1663/5/16384 blk 442</div><div>rmgr: Heap len (rec/tot): 63/ 63, tx: 731, lsn: 0/01039188, prev 0/01039148, desc: INSERT off: 106, flags: 0x00, blkref #0: rel 1663/5/16384 blk 442</div><div>rmgr: Heap len (rec/tot): 63/ 63, tx: 731, lsn: 0/010391C8, prev 0/01039188, desc: INSERT off: 107, flags: 0x00, blkref #0: rel 1663/5/16384 blk 442</div><div>rmgr: Heap len (rec/tot): 63/ 63, tx: 731, lsn: 0/01039208, prev 0/010391C8, desc: INSERT off: 108, flags: 0x00, blkref #0: rel 1663/5/16384 blk 442</div><div>rmgr: Transaction len (rec/tot): 34/ 34, tx: 731, lsn: 0/01039248, prev 0/01039208, desc: COMMIT 2024-08-08 12:52:05.194389 +05</div><div>rmgr: XLOG len (rec/tot): 114/ 114, tx: 0, lsn: 0/01039270, prev 0/01039248, desc: CHECKPOINT_SHUTDOWN redo 0/1039270; tli 1; prev tli 1; fpw true; xid 0:732; oid 16387; multi 1; offset 0; oldest xid 722 in DB 1; oldest multi 1 in DB 1; oldest/newest commit timestamp xid: 0/0; oldest running xid 0; shutdown</div><div>+ pg_waldump -p standby1 000000020000000000000010</div><div>+ tail -n 10</div><div>pg_waldump: error: error in WAL record at 0/103B2F0: invalid record length at 0/103B368: expected at least 24, got 0</div><div>rmgr: Heap len (rec/tot): 63/ 63, tx: 731, lsn: 0/010391C8, prev 0/01039188, desc: INSERT off: 107, flags: 0x00, blkref #0: rel 1663/5/16384 blk 442</div><div>rmgr: Heap len (rec/tot): 63/ 63, tx: 731, lsn: 0/01039208, prev 0/010391C8, desc: INSERT off: 108, flags: 0x00, blkref #0: rel 1663/5/16384 blk 442</div><div>rmgr: Transaction len (rec/tot): 34/ 34, tx: 731, lsn: 0/01039248, prev 0/01039208, desc: COMMIT 2024-08-08 12:52:05.194389 +05</div><div>rmgr: XLOG len (rec/tot): 114/ 114, tx: 0, lsn: 0/01039270, prev 0/01039248, desc: CHECKPOINT_SHUTDOWN redo 0/1039270; tli 1; prev tli 1; fpw true; xid 0:732; oid 16387; multi 1; offset 0; oldest xid 722 in DB 1; oldest multi 1 in DB 1; oldest/newest commit timestamp xid: 0/0; oldest running xid 0; shutdown</div><div>rmgr: XLOG len (rec/tot): 42/ 42, tx: 0, lsn: 0/010392E8, prev 0/01039270, desc: END_OF_RECOVERY tli 2; prev tli 1; time 2024-08-08 12:52:05.946660 +05</div><div>rmgr: Standby len (rec/tot): 50/ 50, tx: 0, lsn: 0/01039318, prev 0/010392E8, desc: RUNNING_XACTS nextXid 732 latestCompletedXid 731 oldestRunningXid 732</div><div>rmgr: XLOG len (rec/tot): 114/ 114, tx: 0, lsn: 0/01039350, prev 0/01039318, desc: CHECKPOINT_ONLINE redo 0/1039318; tli 2; prev tli 2; fpw true; xid 0:732; oid 16387; multi 1; offset 0; oldest xid 722 in DB 1; oldest multi 1 in DB 1; oldest/newest commit timestamp xid: 0/0; oldest running xid 732; online</div><div>rmgr: XLOG len (rec/tot): 49/ 885, tx: 0, lsn: 0/010393C8, prev 0/01039350, desc: FPI_FOR_HINT , blkref #0: rel 1663/5/1259 blk 0 FPW</div><div>rmgr: XLOG len (rec/tot): 49/ 7061, tx: 0, lsn: 0/01039740, prev 0/010393C8, desc: FPI_FOR_HINT , blkref #0: rel 1663/5/1249 blk 17 FPW</div><div>rmgr: XLOG len (rec/tot): 114/ 114, tx: 0, lsn: 0/0103B2F0, prev 0/01039740, desc: CHECKPOINT_SHUTDOWN redo 0/103B2F0; tli 2; prev tli 2; fpw true; xid 0:732; oid 16387; multi 1; offset 0; oldest xid 722 in DB 1; oldest multi 1 in DB 1; oldest/newest commit timestamp xid: 0/0; oldest running xid 0; shutdown</div><div>+ pg_waldump -p standby2 000000010000000000000010</div><div>+ tail -n 10</div><div>pg_waldump: error: error in WAL record at 0/1041348: invalid record length at 0/1041370: expected at least 24, got 0</div><div>rmgr: Heap len (rec/tot): 63/ 63, tx: 732, lsn: 0/01041108, prev 0/010410C8, desc: INSERT off: 214, flags: 0x00, blkref #0: rel 1663/5/16384 blk 443</div><div>rmgr: Heap len (rec/tot): 63/ 63, tx: 732, lsn: 0/01041148, prev 0/01041108, desc: INSERT off: 215, flags: 0x00, blkref #0: rel 1663/5/16384 blk 443</div><div>rmgr: Heap len (rec/tot): 63/ 63, tx: 732, lsn: 0/01041188, prev 0/01041148, desc: INSERT off: 216, flags: 0x00, blkref #0: rel 1663/5/16384 blk 443</div><div>rmgr: Heap len (rec/tot): 63/ 63, tx: 732, lsn: 0/010411C8, prev 0/01041188, desc: INSERT off: 217, flags: 0x00, blkref #0: rel 1663/5/16384 blk 443</div><div>rmgr: Heap len (rec/tot): 63/ 63, tx: 732, lsn: 0/01041208, prev 0/010411C8, desc: INSERT off: 218, flags: 0x00, blkref #0: rel 1663/5/16384 blk 443</div><div>rmgr: Heap len (rec/tot): 63/ 63, tx: 732, lsn: 0/01041248, prev 0/01041208, desc: INSERT off: 219, flags: 0x00, blkref #0: rel 1663/5/16384 blk 443</div><div>rmgr: Heap len (rec/tot): 63/ 63, tx: 732, lsn: 0/01041288, prev 0/01041248, desc: INSERT off: 220, flags: 0x00, blkref #0: rel 1663/5/16384 blk 443</div><div>rmgr: Heap len (rec/tot): 63/ 63, tx: 732, lsn: 0/010412C8, prev 0/01041288, desc: INSERT off: 221, flags: 0x00, blkref #0: rel 1663/5/16384 blk 443</div><div>rmgr: Heap len (rec/tot): 63/ 63, tx: 732, lsn: 0/01041308, prev 0/010412C8, desc: INSERT off: 222, flags: 0x00, blkref #0: rel 1663/5/16384 blk 443</div><div>rmgr: Transaction len (rec/tot): 34/ 34, tx: 732, lsn: 0/01041348, prev 0/01041308, desc: COMMIT 2024-08-08 12:52:06.085591 +05</div><div>+ pg_rewind --progress --debug '--source-pgdata=standby1' '--target-pgdata=standby2'</div><div>pg_rewind: Source timeline history:</div><div>pg_rewind: Target timeline history:</div><div>pg_rewind: 1: 0/0 - 0/0</div><div>pg_rewind: servers diverged at WAL location 0/10392E8 on timeline 1</div><div>pg_rewind: no rewind required</div><div>+ cat</div><div>+ touch standby2/standby.signal</div><div>+ pg_ctl -w -D standby1 start -o '-p 5001'</div><div>waiting for server to start....2024-08-08 12:52:06.432 +05 [2044] DEBUG: registering background worker "logical replication launcher"</div><div>2024-08-08 12:52:06.432 +05 [2044] DEBUG: mmap(146800640) with MAP_HUGETLB failed, huge pages disabled: Out of memory</div><div>2024-08-08 12:52:06.436 +05 [2044] DEBUG: dynamic shared memory system will support 674 segments</div><div>2024-08-08 12:52:06.436 +05 [2044] DEBUG: created dynamic shared memory control segment 8407908 (26976 bytes)</div><div>2024-08-08 12:52:06.436 +05 [2044] DEBUG: max_safe_fds = 986, usable_fds = 1000, already_open = 4</div><div>2024-08-08 12:52:06.436 +05 [2044] LOG: redirecting log output to logging collector process</div><div>2024-08-08 12:52:06.436 +05 [2044] HINT: Future log output will appear in directory "log".</div><div> done</div><div>server started</div><div>+ pg_ctl -w -D standby2 start -o '-p 5002'</div><div>waiting for server to start....2024-08-08 12:52:06.539 +05 [2054] DEBUG: registering background worker "logical replication launcher"</div><div>2024-08-08 12:52:06.539 +05 [2054] DEBUG: mmap(146800640) with MAP_HUGETLB failed, huge pages disabled: Out of memory</div><div>2024-08-08 12:52:06.542 +05 [2054] DEBUG: dynamic shared memory system will support 674 segments</div><div>2024-08-08 12:52:06.542 +05 [2054] DEBUG: created dynamic shared memory control segment 615788588 (26976 bytes)</div><div>2024-08-08 12:52:06.542 +05 [2054] DEBUG: max_safe_fds = 986, usable_fds = 1000, already_open = 4</div><div>2024-08-08 12:52:06.543 +05 [2054] LOG: redirecting log output to logging collector process</div><div>2024-08-08 12:52:06.543 +05 [2054] HINT: Future log output will appear in directory "log".</div><div> done</div><div>server started</div><div>+ psql -d postgres -a --no-psqlrc -p 5001</div><div>select count(*) from a;</div><div> count </div><div>--------</div><div> 100000</div><div>(1 row)</div><div> </div><div>+ psql -d postgres -a --no-psqlrc -p 5002</div><div>select count(*) from a;</div><div> count </div><div>--------</div><div> 100340</div><div>(1 row)</div><div> </div><div>+ pg_ctl -w -D standby1 stop -m fast</div><div>waiting for server to shut down....2024-08-08 12:52:06.792 +05 [2045] DEBUG: logger shutting down</div><div> done</div><div>server stopped</div><div>+ pg_ctl -w -D standby2 stop -m fast</div><div>waiting for server to shut down....2024-08-08 12:52:06.830 +05 [2055] DEBUG: logger shutting down</div><div> done</div><div>server stopped</div></div><div>```</div><div>postgres is compiled from last REL_16_STABLE branch</div></div><div><br /></div><div><br /></div><div>08.08.2024, 12:03, "Heikki Linnakangas" <hlinnaka(at)iki(dot)fi>:</div><blockquote><p>On 07/08/2024 15:19, PG Bug reporting form wrote:<br /></p><blockquote class="210e7a848e8fcb45wmi-quote"> Sometimes pg_rewind mistakenly assumes that nothing needs to be done, which<br /> results in the replica having data that is not on the master.<br /> <br /> ...<br /> + pg_rewind --progress --debug '--source-pgdata=standby1'<br /> '--target-pgdata=standby2'<br /> pg_rewind: Source timeline history:<br /> pg_rewind: Target timeline history:<br /> pg_rewind: 1: 0/0 - 0/0<br /> pg_rewind: servers diverged at WAL location 0/10392E8 on timeline 1<br /> pg_rewind: no rewind required<br /><br /> ...<br /> Same query<br /> + psql -d postgres -a --no-psqlrc -p 5001<br /> select count(*) from a;<br /> count<br /> --------<br /> 100000<br /> (1 row)<br /> <br /> + echo 'Different results'<br /> Different results<br /> + psql -d postgres -a --no-psqlrc -p 5002<br /> select count(*) from a;<br /> count<br /> --------<br /> 100340<br /> (1 row)<br /> <br /> + echo 'Stop standby1'<br /> Stop standby1<br /> + pg_ctl -w -D standby1 stop -m fast<br /> waiting for server to shut down....<span class="1f1ea193f6735cf0wmi-callto">2024-08-07 17</span>:15:03.077 +05 [19703]<br /> DEBUG: logger shutting down<br /> done<br /> server stopped<br /> + echo 'Stop standby2'<br /> Stop standby2<br /> + pg_ctl -w -D standby2 stop -m fast<br /> waiting for server to shut down....<span class="1f1ea193f6735cf0wmi-callto">2024-08-07 17</span>:15:03.117 +05 [19713]<br /> DEBUG: logger shutting down<br /> done<br /> server stopped<br /> ```<br /></blockquote><p><br />I cannot reproduce this. On my laptop, the reported "servers diverged at <br />WAL location" is always higher, and it performs rewind as expected:<br /><br />g_rewind: Source timeline history:<br />pg_rewind: Target timeline history:<br />pg_rewind: 1: 0/0 - 0/0<br />pg_rewind: servers diverged at WAL location 0/1138F00 on timeline 1<br />pg_rewind: rewinding from last common checkpoint at 0/1138E88 on timeline 1<br /><br />However while looking at the code, I noticed that the debug-output of <br />the Source timeline history is broken. See attached patch to fix it. <br />With that fix, I get:<br /><br />pg_rewind: Source timeline history:<br />pg_rewind: 1: 0/0 - 0/1138F00<br />pg_rewind: 2: 0/1138F00 - 0/0<br />pg_rewind: Target timeline history:<br />pg_rewind: 1: 0/0 - 0/0<br />pg_rewind: 0 a: 1 0/0-0/1138F00 - b: 1 0/0-0/0<br />pg_rewind: servers diverged at WAL location 0/1138F00 on timeline 1<br />pg_rewind: rewinding from last common checkpoint at 0/1138E88 on timeline 1<br /><br />That just fixes the debugging output, though, it doesn't change what it <br />actually does.<br /><br />Can you try the attached fix show the output you get with that please? <br />Or alternatively, show the contents of the <br />standby1/pg_wal/<span class="1f1ea193f6735cf0wmi-callto">00000002</span>.history file.<br /><br /></p><span class="f55bbb4eeef208e8wmi-sign">-- <br />Heikki Linnakangas<br />Neon (<a href="https://neon.tech/">https://neon.tech</a>)<br /></span></blockquote>