RE: Perform streaming logical transactions by background workers and parallel apply

From: "wangw(dot)fnst(at)fujitsu(dot)com" <wangw(dot)fnst(at)fujitsu(dot)com>
To: Peter Smith <smithpb2250(at)gmail(dot)com>
Cc: Amit Kapila <amit(dot)kapila16(at)gmail(dot)com>, Masahiko Sawada <sawada(dot)mshk(at)gmail(dot)com>, "shiy(dot)fnst(at)fujitsu(dot)com" <shiy(dot)fnst(at)fujitsu(dot)com>, "houzj(dot)fnst(at)fujitsu(dot)com" <houzj(dot)fnst(at)fujitsu(dot)com>, PostgreSQL Hackers <pgsql-hackers(at)lists(dot)postgresql(dot)org>
Subject: RE: Perform streaming logical transactions by background workers and parallel apply
Date: 2022-07-07 10:20:45
Message-ID: OS3PR01MB627594D75E870BBB45E2E80A9E839@OS3PR01MB6275.jpnprd01.prod.outlook.com
Views: Raw Message | Whole Thread | Download mbox | Resend email
Thread:
Lists: pgsql-hackers

On Fri, Jul 7, 2022 at 11:44 AM I wrote:
> Attach the new patches.

I found a failure on CFbot [1], which after investigation I think is due to my
previous modification (see response to #1.10 in [2]).

For a streaming transaction, if we failed in the first chunk of streamed
changes for this transaction in the apply background worker, we will set the
status of this apply background worker to APPLY_BGWORKER_EXIT.
And at the same time, main apply worker obtains apply background worker
in the function apply_bgworker_find when processing the second chunk of
streamed changes for this transaction, the status of apply background worker
is APPLY_BGWORKER_EXIT. So the following assertion will fail:
```
Assert(status == APPLY_BGWORKER_BUSY);
```

To fix this, before invoking function assert, I try to detect the failure of
apply background worker. If the status is APPLY_BGWORKER_EXIT, then exit with
an error.

I also made some other small improvements.

Attach the new patches.

[1] - https://cirrus-ci.com/task/6383178511286272?logs=test_world#L2636
[2] - https://www.postgresql.org/message-id/OS3PR01MB62755C6C9A75EB09F7218B589E839%40OS3PR01MB6275.jpnprd01.prod.outlook.com

Regards,
Wang wei

Attachment Content-Type Size
v16-0001-Perform-streaming-logical-transactions-by-backgr.patch application/octet-stream 105.9 KB
v16-0002-Test-streaming-parallel-option-in-tap-test.patch application/octet-stream 69.1 KB
v16-0003-Add-some-checks-before-using-apply-background-wo.patch application/octet-stream 36.6 KB
v16-0004-Retry-to-apply-streaming-xact-only-in-apply-work.patch application/octet-stream 26.6 KB

In response to

Responses

Browse pgsql-hackers by date

  From Date Subject
Next Message Alexander Korotkov 2022-07-07 10:43:55 Re: Add red-black tree missing comparison searches
Previous Message vignesh C 2022-07-07 10:14:37 Re: explain analyze rows=%.0f