RE: bdr replication breaks down

Hi again,

So I checked the x_logs and found the following lines.

It seems like it faild at commiting a long transaction…

I am still not sure what the pg_temp30 schema is used for.

But I was able to “fix” it with

select bdr.skip_changes_upto('6208877715678412212', 1, 2942745,
'7D/2E314530');

I guess the listed transaction is now missing in some of the nodes.

rmgr: Standby len (rec/tot): 28/ 60, tx: 0, lsn:
7D/2E313238, prev 7D/2E313208, bkp: 0000, desc: running xacts: nextXid
106791716 latestCompletedXid 106791714 oldestRunningXid 106791715; 1 xacts:
106791715

rmgr: Heap2 len (rec/tot): 34/ 66, tx: 106791715, lsn:
7D/2E313278, prev 7D/2E313238, bkp: 0000, desc: new_cid: rel
1663/2942745/12834; tid 14/1; cmin: 4294967295, cmax: 0, combo: 4294967295

rmgr: Heap len (rec/tot): 26/ 58, tx: 106791715, lsn:
7D/2E3132C0, prev 7D/2E313278, bkp: 0000, desc: delete: rel
1663/2942745/12834; tid 14/1 KEYS_UPDATED

rmgr: Heap2 len (rec/tot): 34/ 66, tx: 106791715, lsn:
7D/2E313300, prev 7D/2E3132C0, bkp: 0000, desc: new_cid: rel
1663/2942745/12877; tid 65/82; cmin: 4294967295, cmax: 0, combo: 4294967295

rmgr: Heap len (rec/tot): 26/ 58, tx: 106791715, lsn:
7D/2E313348, prev 7D/2E313300, bkp: 0000, desc: delete: rel
1663/2942745/12877; tid 65/82 KEYS_UPDATED

rmgr: Heap2 len (rec/tot): 34/ 66, tx: 106791715, lsn:
7D/2E313388, prev 7D/2E313348, bkp: 0000, desc: new_cid: rel
1663/2942745/12877; tid 65/81; cmin: 4294967295, cmax: 0, combo: 4294967295

rmgr: Heap len (rec/tot): 26/ 58, tx: 106791715, lsn:
7D/2E3133D0, prev 7D/2E313388, bkp: 0000, desc: delete: rel
1663/2942745/12877; tid 65/81 KEYS_UPDATED

rmgr: Heap2 len (rec/tot): 34/ 66, tx: 106791715, lsn:
7D/2E313410, prev 7D/2E3133D0, bkp: 0000, desc: new_cid: rel
1663/2942745/12877; tid 65/80; cmin: 4294967295, cmax: 0, combo: 4294967295

rmgr: Heap len (rec/tot): 26/ 58, tx: 106791715, lsn:
7D/2E313458, prev 7D/2E313410, bkp: 0000, desc: delete: rel
1663/2942745/12877; tid 65/80 KEYS_UPDATED

rmgr: Heap2 len (rec/tot): 34/ 66, tx: 106791715, lsn:
7D/2E313498, prev 7D/2E313458, bkp: 0000, desc: new_cid: rel
1663/2942745/12877; tid 65/79; cmin: 4294967295, cmax: 0, combo: 4294967295

rmgr: Heap len (rec/tot): 26/ 58, tx: 106791715, lsn:
7D/2E3134E0, prev 7D/2E313498, bkp: 0000, desc: delete: rel
1663/2942745/12877; tid 65/79 KEYS_UPDATED

rmgr: Heap2 len (rec/tot): 34/ 66, tx: 106791715, lsn:
7D/2E313520, prev 7D/2E3134E0, bkp: 0000, desc: new_cid: rel
1663/2942745/12877; tid 65/78; cmin: 4294967295, cmax: 0, combo: 4294967295

rmgr: Heap len (rec/tot): 26/ 58, tx: 106791715, lsn:
7D/2E313568, prev 7D/2E313520, bkp: 0000, desc: delete: rel
1663/2942745/12877; tid 65/78 KEYS_UPDATED

rmgr: Heap2 len (rec/tot): 34/ 66, tx: 106791715, lsn:
7D/2E3135A8, prev 7D/2E313568, bkp: 0000, desc: new_cid: rel
1663/2942745/12877; tid 65/77; cmin: 4294967295, cmax: 0, combo: 4294967295

rmgr: Heap len (rec/tot): 26/ 58, tx: 106791715, lsn:
7D/2E3135F0, prev 7D/2E3135A8, bkp: 0000, desc: delete: rel
1663/2942745/12877; tid 65/77 KEYS_UPDATED

rmgr: Heap2 len (rec/tot): 34/ 66, tx: 106791715, lsn:
7D/2E313630, prev 7D/2E3135F0, bkp: 0000, desc: new_cid: rel
1663/2942745/12877; tid 65/76; cmin: 4294967295, cmax: 0, combo: 4294967295

rmgr: Heap len (rec/tot): 26/ 58, tx: 106791715, lsn:
7D/2E313678, prev 7D/2E313630, bkp: 0000, desc: delete: rel
1663/2942745/12877; tid 65/76 KEYS_UPDATED

rmgr: Heap2 len (rec/tot): 34/ 66, tx: 106791715, lsn:
7D/2E3136B8, prev 7D/2E313678, bkp: 0000, desc: new_cid: rel
1663/2942745/12877; tid 65/75; cmin: 4294967295, cmax: 0, combo: 4294967295

rmgr: Heap len (rec/tot): 26/ 58, tx: 106791715, lsn:
7D/2E313700, prev 7D/2E3136B8, bkp: 0000, desc: delete: rel
1663/2942745/12877; tid 65/75 KEYS_UPDATED

rmgr: Heap2 len (rec/tot): 34/ 66, tx: 106791715, lsn:
7D/2E313740, prev 7D/2E313700, bkp: 0000, desc: new_cid: rel
1663/2942745/12877; tid 65/74; cmin: 4294967295, cmax: 0, combo: 4294967295

rmgr: Heap len (rec/tot): 26/ 58, tx: 106791715, lsn:
7D/2E313788, prev 7D/2E313740, bkp: 0000, desc: delete: rel
1663/2942745/12877; tid 65/74 KEYS_UPDATED

rmgr: Heap2 len (rec/tot): 34/ 66, tx: 106791715, lsn:
7D/2E3137C8, prev 7D/2E313788, bkp: 0000, desc: new_cid: rel
1663/2942745/12747; tid 12/10; cmin: 4294967295, cmax: 1, combo: 4294967295

rmgr: Heap len (rec/tot): 26/ 58, tx: 106791715, lsn:
7D/2E313810, prev 7D/2E3137C8, bkp: 0000, desc: delete: rel
1663/2942745/12747; tid 12/10 KEYS_UPDATED

rmgr: Heap2 len (rec/tot): 34/ 66, tx: 106791715, lsn:
7D/2E313850, prev 7D/2E313810, bkp: 0000, desc: new_cid: rel
1663/2942745/12877; tid 65/72; cmin: 4294967295, cmax: 1, combo: 4294967295

rmgr: Heap len (rec/tot): 26/ 58, tx: 106791715, lsn:
7D/2E313898, prev 7D/2E313850, bkp: 0000, desc: delete: rel
1663/2942745/12877; tid 65/72 KEYS_UPDATED

rmgr: Heap2 len (rec/tot): 34/ 66, tx: 106791715, lsn:
7D/2E3138D8, prev 7D/2E313898, bkp: 0000, desc: new_cid: rel
1663/2942745/12747; tid 12/9; cmin: 4294967295, cmax: 2, combo: 4294967295

rmgr: Heap len (rec/tot): 26/ 58, tx: 106791715, lsn:
7D/2E313920, prev 7D/2E3138D8, bkp: 0000, desc: delete: rel
1663/2942745/12747; tid 12/9 KEYS_UPDATED

rmgr: Heap2 len (rec/tot): 34/ 66, tx: 106791715, lsn:
7D/2E313960, prev 7D/2E313920, bkp: 0000, desc: new_cid: rel
1663/2942745/12877; tid 65/71; cmin: 4294967295, cmax: 2, combo: 4294967295

rmgr: Heap len (rec/tot): 26/ 58, tx: 106791715, lsn:
7D/2E3139A8, prev 7D/2E313960, bkp: 0000, desc: delete: rel
1663/2942745/12877; tid 65/71 KEYS_UPDATED

rmgr: Heap2 len (rec/tot): 34/ 66, tx: 106791715, lsn:
7D/2E3139E8, prev 7D/2E3139A8, bkp: 0000, desc: new_cid: rel
1663/2942745/12759; tid 74/33; cmin: 4294967295, cmax: 3, combo: 4294967295

rmgr: Heap len (rec/tot): 26/ 58, tx: 106791715, lsn:
7D/2E313A30, prev 7D/2E3139E8, bkp: 0000, desc: delete: rel
1663/2942745/12759; tid 74/33 KEYS_UPDATED

rmgr: Heap2 len (rec/tot): 34/ 66, tx: 106791715, lsn:
7D/2E313A70, prev 7D/2E313A30, bkp: 0000, desc: new_cid: rel
1663/2942745/12759; tid 74/34; cmin: 4294967295, cmax: 3, combo: 4294967295

rmgr: Heap len (rec/tot): 26/ 58, tx: 106791715, lsn:
7D/2E313AB8, prev 7D/2E313A70, bkp: 0000, desc: delete: rel
1663/2942745/12759; tid 74/34 KEYS_UPDATED

rmgr: Heap2 len (rec/tot): 34/ 66, tx: 106791715, lsn:
7D/2E313AF8, prev 7D/2E313AB8, bkp: 0000, desc: new_cid: rel
1663/2942745/12759; tid 74/35; cmin: 4294967295, cmax: 3, combo: 4294967295

rmgr: Heap len (rec/tot): 26/ 58, tx: 106791715, lsn:
7D/2E313B40, prev 7D/2E313AF8, bkp: 0000, desc: delete: rel
1663/2942745/12759; tid 74/35 KEYS_UPDATED

rmgr: Heap2 len (rec/tot): 34/ 66, tx: 106791715, lsn:
7D/2E313B80, prev 7D/2E313B40, bkp: 0000, desc: new_cid: rel
1663/2942745/12770; tid 14/33; cmin: 4294967295, cmax: 3, combo: 4294967295

rmgr: Heap len (rec/tot): 26/ 58, tx: 106791715, lsn:
7D/2E313BC8, prev 7D/2E313B80, bkp: 0000, desc: delete: rel
1663/2942745/12770; tid 14/33 KEYS_UPDATED

rmgr: Heap2 len (rec/tot): 34/ 66, tx: 106791715, lsn:
7D/2E313C08, prev 7D/2E313BC8, bkp: 0000, desc: new_cid: rel
1663/2942745/12877; tid 65/73; cmin: 4294967295, cmax: 3, combo: 4294967295

rmgr: Heap len (rec/tot): 26/ 58, tx: 106791715, lsn:
7D/2E313C50, prev 7D/2E313C08, bkp: 0000, desc: delete: rel
1663/2942745/12877; tid 65/73 KEYS_UPDATED

rmgr: Heap2 len (rec/tot): 34/ 66, tx: 106791715, lsn:
7D/2E313C90, prev 7D/2E313C50, bkp: 0000, desc: new_cid: rel 1664/0/12906;
tid 1/49; cmin: 4294967295, cmax: 3, combo: 4294967295

rmgr: Heap len (rec/tot): 26/ 58, tx: 106791715, lsn:
7D/2E313CD8, prev 7D/2E313C90, bkp: 0000, desc: delete: rel 1664/0/12906;
tid 1/49 KEYS_UPDATED

rmgr: Heap len (rec/tot): 160/ 1660, tx: 106791715, lsn:
7D/2E313D18, prev 7D/2E313CD8, bkp: 1000, desc: insert: rel
1663/2942745/2943410; tid 0/7

rmgr: Transaction len (rec/tot): 352/ 384, tx: 106791715, lsn:
7D/2E3143B0, prev 7D/2E313D18, bkp: 0000, desc: commit: 2019-01-28
01:12:34.348577 PST; inval msgs: catcache 7 catcache 6 catcache 7 catcache
6 catcache 7 catcache 6 catcache 46 catcache 45 catcache 64 catcache 63
catcache 64 catcache 63 catcache 49 relcache 23783862 snapshot 2608
snapshot 1214 snapshot 2608 snapshot 2608 relcache 23783862 snapshot 2608

rmgr: Standby len (rec/tot): 44/ 76, tx: 0, lsn:
7D/2E314530, prev 7D/2E3143B0, bkp: 0000, desc: nontransactional message
with 35 bytes

*From:* Daniel Fink (PDF) [mailto:daniel(dot)fink(at)pdf(dot)com]
*Sent:* Monday, February 11, 2019 11:19 AM
*To:* pgsql-general(at)lists(dot)postgresql(dot)org
*Subject:* bdr replication breaks down

Hi all,

I have a bdr replication setup that worked fine until recently.

But it seems like one command cannot be replicated, and it breaks down the
whole process on all hosts where it is sent to.

I appended part of the log file, it gets repeated over and over.

Can I somehow omit this command and try if the next one works?

Best Regards,

Daniel Fink

< 2019-02-08 00:00:01.658 CET >ERROR: schema "pg_temp_30" does not exist

< 2019-02-08 00:00:01.658 CET >CONTEXT: apply QUEUED_DROP in commit
7D/2E3143B0, xid 106791715 commited at 2019-01-28 10:12:34.348577+01
(action #2) from node (6208877715678412212,1,2942745)

< 2019-02-08 00:00:01.661 CET >LOG: worker process: bdr
(6449651545875285115,1,16388,)->bdr (6208877715678412212,1, (PID 27640)
exited with exit code 1

< 2019-02-08 00:00:06.666 CET >LOG: starting background worker process
"bdr (6449651545875285115,1,16388,)->bdr (6208877715678412212,1,"

< 2019-02-08 00:00:07.641 CET >ERROR: schema "pg_temp_30" does not exist

< 2019-02-08 00:00:07.641 CET >CONTEXT: apply QUEUED_DROP in commit
7D/2E3143B0, xid 106791715 commited at 2019-01-28 10:12:34.348577+01
(action #2) from node (6208877715678412212,1,2942745)

< 2019-02-08 00:00:07.644 CET >LOG: worker process: bdr
(6449651545875285115,1,16388,)->bdr (6208877715678412212,1, (PID 27645)
exited with exit code 1

< 2019-02-08 00:00:12.649 CET >LOG: starting background worker process
"bdr (6449651545875285115,1,16388,)->bdr (6208877715678412212,1,"

< 2019-02-08 00:00:13.633 CET >ERROR: schema "pg_temp_30" does not exist

< 2019-02-08 00:00:13.633 CET >CONTEXT: apply QUEUED_DROP in commit
7D/2E3143B0, xid 106791715 commited at 2019-01-28 10:12:34.348577+01
(action #2) from node (6208877715678412212,1,2942745)

*DANIEL FINK**Senior Software Engineer*

*tel* (+49) 89.767062.20
*fax*(+49) 89.767062.11
email daniel(dot)fink(at)pdf(dot)com

*PDF Solutions GmbH*
* (**a PDF Solutions Company)*
Managing Director: Kimon Michaels
Schwanthalerstr. 10
D-80336 München, Germany

München HRB 87307
DE 128214899

*www.pdf.com <http://www.pdf.com/>*

--
This message may contain confidential and privileged information. If it has
been sent to you in error, please reply to advise the sender of the error
and then immediately permanently delete it and all attachments to it from
your systems. If you are not the intended recipient, do not read, copy,
disclose or otherwise use this message or any attachments to it. The sender
disclaims any liability for such unauthorized use. PLEASE NOTE that all
incoming e-mails sent to PDF e-mail accounts will be archived and may be
scanned by us and/or by external service providers to detect and prevent
threats to our systems, investigate illegal or inappropriate behavior,
and/or eliminate unsolicited promotional e-mails (“spam”). If you have any
concerns about this process, please contact us at legal(dot)department(at)pdf(dot)com
<mailto:legal(dot)department(at)pdf(dot)com>.