RE: Perform streaming logical transactions by background workers and parallel apply

From: "wangw(dot)fnst(at)fujitsu(dot)com" <wangw(dot)fnst(at)fujitsu(dot)com>
To: Amit Kapila <amit(dot)kapila16(at)gmail(dot)com>, Masahiko Sawada <sawada(dot)mshk(at)gmail(dot)com>
Cc: Peter Smith <smithpb2250(at)gmail(dot)com>, "shiy(dot)fnst(at)fujitsu(dot)com" <shiy(dot)fnst(at)fujitsu(dot)com>, "houzj(dot)fnst(at)fujitsu(dot)com" <houzj(dot)fnst(at)fujitsu(dot)com>, PostgreSQL Hackers <pgsql-hackers(at)lists(dot)postgresql(dot)org>
Subject: RE: Perform streaming logical transactions by background workers and parallel apply
Date: 2022-06-14 03:37:04
Message-ID: OS3PR01MB62756BD9482EB6BB1CA4CD4D9EAA9@OS3PR01MB6275.jpnprd01.prod.outlook.com
Views: Raw Message | Whole Thread | Download mbox | Resend email
Thread:
Lists: pgsql-hackers

On Wed, Jun 8, 2022 3:13 PM I wrote:
> Attach the new patches.(only changed 0001 and 0003)

I tried to improve the patches by following points:

1. Initialize variable include_abort_lsn to false. It reports a warning in
cfbot. (see patch v10-0001)
BTW, I merged the patch that added the new GUC (see v9-0004) into patch 0001.

2. Because of the improvement #2 in [1], the foreign key could not be detected
when checking trigger function. So added additional checks for the foreign key.
(see patch 0004)

3. Adding a check for the partition table when trying to apply changes in the
apply background worker. (see patch 0004)
In additional, the partition cache map on subscriber have several bugs (see
thread [2]). Because patch 0004 is developed based on the patches in [2], so I
merged the patches(v4-0001~v4-0003) in [2] into a temporary patch 0003 here.
After the patches in [2] is committed, I will delete patch 0003 and rebase
patch 0004.

4. Improve constraint checking in a separate patch as suggested by Amit-san in
[3] #6.(see patch 0005)
I added a new field "bool subretry" in catalog pg_subscription. I use this
field to indicate whether the transaction that we are going to process has
failed before.
If apply worker/bgworker was exit with an error, this field will be set to
true; If we successfully apply a transaction, this field will be set to false.
If we retry to apply a streaming transaction, whether the user sets the
streaming option to "on" or "apply", we will apply the transaction in the apply
worker.

Attach the new patches.
Only changed patches 0001, 0004 and added new separate patch 0005.

[1] - https://www.postgresql.org/message-id/OS3PR01MB6275208A2F8ED832710F65E09EA49%40OS3PR01MB6275.jpnprd01.prod.outlook.com
[2] - https://www.postgresql.org/message-id/flat/OSZPR01MB6310F46CD425A967E4AEF736FDA49%40OSZPR01MB6310.jpnprd01.prod.outlook.com
[3] - https://www.postgresql.org/message-id/CAA4eK1Jt08SYbRt_-rbSWNg%3DX9-m8%2BRdP5PosfnQgyF-z8bkxQ%40mail.gmail.com

Regards,
Wang wei

Attachment Content-Type Size
v10-0001-Perform-streaming-logical-transactions-by-backgr.patch application/octet-stream 98.2 KB
v10-0002-Test-streaming-apply-option-in-tap-test.patch application/octet-stream 68.9 KB
v10-0003-The-v4-patch-0001-0003-in-thread-1.patch application/octet-stream 17.2 KB
v10-0004-Add-some-checks-before-using-apply-background-wo.patch application/octet-stream 29.1 KB
v10-0005-Retry-to-apply-streaming-xact-only-in-apply-work.patch application/octet-stream 20.4 KB

In response to

Responses

Browse pgsql-hackers by date

  From Date Subject
Next Message houzj.fnst@fujitsu.com 2022-06-14 03:40:42 RE: Skipping schema changes in publication
Previous Message huangning290@yahoo.com 2022-06-14 03:35:52 GIN index partial match