Re: BUG #16760: Standby database missed records for at least 1 table

Here’s output you requested
lp | lp_off | lp_flags | lp_len | t_xmin | t_xmax | t_field3 | t_ctid | infomask | infomask2 | t_hoff | t_bits
----+--------+----------+--------+-----------+--------+----------+-----------+----------+-----------+--------+------------------------------------------------------------------
6 | 2616 | 1 | 227 | 388763235 | 0 | 80 | (1982,6) | a03 | 3e | 32 | 1111100100110000000000000000001000000101010000010101011111000100
7 | 2304 | 1 | 307 | 388763235 | 0 | 0 | (1982,7) | a03 | 3e | 32 | 1111110100110000000000001000001000000101011000010101011111000100
32 | 2072 | 1 | 227 | 388763235 | 0 | 0 | (1982,32) | a03 | 3e | 32 | 1111100100110000000000000000001000000101010000010101011111000100
(3 rows)

From: Euler Taveira <euler(dot)taveira(at)2ndquadrant(dot)com>
Date: Wednesday, December 9, 2020 at 4:48 PM
To: Andriy Bartash <abartash(at)xmatters(dot)com>
Cc: pgsql-bugs(at)lists(dot)postgresql(dot)org <pgsql-bugs(at)lists(dot)postgresql(dot)org>
Subject: Re: BUG #16760: Standby database missed records for at least 1 table
On Tue, 8 Dec 2020 at 21:03, Andriy Bartash <abartash(at)xmatters(dot)com<mailto:abartash(at)xmatters(dot)com>> wrote:
Hi Euler
I inspected pages where I expected to have my data on broken standby.
What was done:

1. Identified the page where was my data (on primary cluster)

select ctid,audit_ev_id from audit_evs_all where audit_ev_id between 221535154 and 221535162;

and it returned
ctid,audit_ev_id
"(1979,3)",221535154
"(1980,13)",221535155
"(1980,14)",221535156
"(1980,28)",221535157
"(1980,29)",221535158
"(1982,6)",221535159
"(1982,7)",221535160
"(1982,32)",221535161
"(1982,33)",221535162

Found that page # 1982 had 4 records on primary (ev_id 221535159, 221535160, 221535161 and 221535162)
If you already have the broken secondary, share the following query results from both servers (requires extension pageinspect):

SELECT lp, lp_off, lp_flags, lp_len, t_xmin, t_xmax, t_field3, t_ctid, to_hex(t_infomask) AS infomask, to_hex(t_infomask2) AS infomask2, t_hoff, t_bits FROM heap_page_items(get_raw_page('audit_evs_all', 1982)) WHERE t_ctid IN('1982,6)', '(1982,7)', '(1982,32)');

--
Euler Taveira http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services