RE: Proposal: Filter irrelevant change before reassemble transactions during logical decoding

From: "Hayato Kuroda (Fujitsu)" <kuroda(dot)hayato(at)fujitsu(dot)com>
To: 'Ajin Cherian' <itsajin(at)gmail(dot)com>, "Zhijie Hou (Fujitsu)" <houzj(dot)fnst(at)fujitsu(dot)com>
Cc: Amit Kapila <amit(dot)kapila16(at)gmail(dot)com>, PostgreSQL Hackers <pgsql-hackers(at)lists(dot)postgresql(dot)org>
Subject: RE: Proposal: Filter irrelevant change before reassemble transactions during logical decoding
Date: 2025-01-31 13:43:18
Message-ID: OSCPR01MB14966F9A206C5191A85433632F5E82@OSCPR01MB14966.jpnprd01.prod.outlook.com
Views: Raw Message | Whole Thread | Download mbox | Resend email
Thread:
Lists: pgsql-hackers

Dear Ajin, Hou,

> - Snapshot construction

I understand the approach that we do not try to filter for all the workloads; do
just best-effort.

I played with your PoC and here are my comments.

1.
Can you add tests for better understanding? I've tried for tes_decoding like attached,
but it couldn't pass the regression test. Cases "stream.sql" and "twophase_stream.sql"
were failed.

2.
I think we can extend the skip mechanism to UPDATE/DELETE/MultiInsert/SpecConfirm.
Regarding the TRUNCATE, I'm not sure we can handle hte TRUNCATE case because the we
can't track RelFileLocator anymore.

3.
Both ReorderBuffer and RelFileLocatorFilterCache have the same lifetime but
RelFileLocatorFilterCache is provided as the global variable; it is quite strange.
Can you somehow avoid this? I considered idea but could not because we do not have
APIs to Unregister the relcacheCallback.

4.
For output plugins which does not cache the catalog, it may not able to do filtering
without the transaction. For them, the early filter may degrade the performance
because it requires to open transactions for every changes.
Is my understanding correct?

Best regards,
Hayato Kuroda
FUJITSU LIMITED

Attachment Content-Type Size
v20250131-0001-Try-to-add-tests.txt text/plain 5.0 KB

In response to

Browse pgsql-hackers by date

  From Date Subject
Next Message Vitaly Davydov 2025-01-31 13:54:17 Re: Issues with 2PC at recovery: CLOG lookups and GlobalTransactionData
Previous Message Alvaro Herrera 2025-01-31 13:40:50 Re: NOT ENFORCED constraint feature