RE: long-standing data loss bug in initial sync of logical replication

From: "Hayato Kuroda (Fujitsu)" <kuroda(dot)hayato(at)fujitsu(dot)com>
To: 'Amit Kapila' <amit(dot)kapila16(at)gmail(dot)com>
Cc: PostgreSQL Hackers <pgsql-hackers(at)lists(dot)postgresql(dot)org>, "Zhijie Hou (Fujitsu)" <houzj(dot)fnst(at)fujitsu(dot)com>, Shlok Kyal <shlok(dot)kyal(dot)oss(at)gmail(dot)com>, vignesh C <vignesh21(at)gmail(dot)com>, Nitin Motiani <nitinmotiani(at)google(dot)com>, Andres Freund <andres(at)anarazel(dot)de>, Masahiko Sawada <sawada(dot)mshk(at)gmail(dot)com>
Subject: RE: long-standing data loss bug in initial sync of logical replication
Date: 2025-03-17 08:37:33
Message-ID: OSCPR01MB14966DE3D5999CF5CAE008200F5DF2@OSCPR01MB14966.jpnprd01.prod.outlook.com
Views: Raw Message | Whole Thread | Download mbox | Resend email
Thread:
Lists: pgsql-hackers

Dear Amit,

> A few comments:
> ==============
> 1.
> +SnapBuildDistributeSnapshotAndInval(SnapBuild *builder, XLogRecPtr
> lsn, TransactionId xid)
> {
> dlist_iter txn_i;
> ReorderBufferTXN *txn;
> + ReorderBufferTXN *curr_txn;
> +
> + curr_txn = ReorderBufferTXNByXid(builder->reorder, xid, false, NULL,
> InvalidXLogRecPtr, false);
>
> The above is used to access invalidations from curr_txn. I am thinking
> about whether it would be better to expose a new function to get
> invalidations for a txn based on xid instead of getting
> ReorderBufferTXN. It would avoid any layering violation and misuse of
> ReorderBufferTXN by other modules.

Sounds reasonable. I introduced new function ReorderBufferGetInvalidations() which
obtains the number of invalidations and its list. ReorderBufferTXN() is not exported
anymore.

> 2. The patch has a lot of tests to verify the same points. Can't we
> have one simple test using SQL API based on what Andres presented in
> an email [1]?

You meant that we need to test only the case reported by the Andres, right? New
version did like that. To make the test faster test was migrated to isolation tester
instead of the TAP test.

> 3. I have made minor changes in the comments in the attached.

Thanks, I included.

PSA new version for PG 14-master. Special thanks for Hou to minimize the test code.

Best regards,
Hayato Kuroda
FUJITSU LIMITED

Attachment Content-Type Size
v19_REL_14-0001-Distribute-invalidatons-if-change-in-cata.patch application/octet-stream 10.9 KB
v19_REL_15-0001-Distribute-invalidatons-if-change-in-cata.patch application/octet-stream 10.9 KB
v19-0001-Distribute-invalidatons-if-change-in-catalog-tab.patch application/octet-stream 11.3 KB
v19_REL_17-0001-Distribute-invalidatons-if-change-in-cata.patch application/octet-stream 11.3 KB
v19_REL_16-0001-Distribute-invalidatons-if-change-in-cata.patch application/octet-stream 11.3 KB

In response to

Responses

Browse pgsql-hackers by date

  From Date Subject
Next Message Zharkov Roman 2025-03-17 08:43:13 Re: plperl version on the meson setup summary screen
Previous Message Ashutosh Bapat 2025-03-17 08:36:50 Re: Enhancing Memory Context Statistics Reporting