RE: Perform streaming logical transactions by background workers and parallel apply

From: "wangw(dot)fnst(at)fujitsu(dot)com" <wangw(dot)fnst(at)fujitsu(dot)com>
To: Amit Kapila <amit(dot)kapila16(at)gmail(dot)com>, "houzj(dot)fnst(at)fujitsu(dot)com" <houzj(dot)fnst(at)fujitsu(dot)com>
Cc: Peter Smith <smithpb2250(at)gmail(dot)com>, Masahiko Sawada <sawada(dot)mshk(at)gmail(dot)com>, Dilip Kumar <dilipbalaut(at)gmail(dot)com>, "shiy(dot)fnst(at)fujitsu(dot)com" <shiy(dot)fnst(at)fujitsu(dot)com>, PostgreSQL Hackers <pgsql-hackers(at)lists(dot)postgresql(dot)org>
Subject: RE: Perform streaming logical transactions by background workers and parallel apply
Date: 2022-12-27 04:54:02
Message-ID: OS3PR01MB6275B61076717E4CE9E079D19EED9@OS3PR01MB6275.jpnprd01.prod.outlook.com
Views: Raw Message | Whole Thread | Download mbox | Resend email
Thread:
Lists: pgsql-hackers

On Mon, Dec 26, 2022 19:51 PM Amit Kapila <amit(dot)kapila16(at)gmail(dot)com> wrote:
> Few comments on 0002 and 0001 patches
> =================================

Thanks for your comments.

> 1.
> + if ($is_parallel)
> + {
> + $node_subscriber->append_conf('postgresql.conf',
> + "log_min_messages = debug1");
> + $node_subscriber->reload;
> + }
> +
> + # Check the subscriber log from now on.
> + $offset = -s $node_subscriber->logfile;
> +
> + $in .= q{
> + BEGIN;
> + INSERT INTO test_tab SELECT i, md5(i::text) FROM
> generate_series(3, 5000) s(i);
>
> How can we guarantee that reload would have taken place before this
> next test? I see that 020_archive_status.pl is executing a query to
> ensure the reload has been taken into consideration. Can we do the
> same?

Agree. Improved as suggested.

> 2. It is not very clear whether converting 017_stream_ddl and
> 019_stream_subxact_ddl_abort adds much value. They seem to be mostly
> testing DDL/DML interaction of publisher side. We can probably check
> the code coverage by removing the parallel version for these two files
> and remove them unless it covers some extra code. If we decide to
> remove parallel version for these two files then we can probably add a
> comment atop these files indicating why we don't have a version that
> parallel option for these tests.

I have checked this and removed the parallel version for these two files.
Also added some comments atop these two test files to explain this.

> 3.
> +# Drop the unique index on the subscriber, now it works.
> +$node_subscriber->safe_psql('postgres', "DROP INDEX idx_tab");
> +
> +# Wait for this streaming transaction to be applied in the apply worker.
> $node_publisher->wait_for_catchup($appname);
>
> $result =
> - $node_subscriber->safe_psql('postgres',
> - "SELECT count(*), count(c), count(d = 999) FROM test_tab");
> -is($result, qq(3334|3334|3334), 'check extra columns contain local defaults');
> + $node_subscriber->safe_psql('postgres', "SELECT count(*) FROM
> test_tab_2");
> +is($result, qq(5001), 'data replicated to subscriber after dropping index');
>
> -# Test the streaming in binary mode
> +# Clean up test data from the environment.
> +$node_publisher->safe_psql('postgres', "TRUNCATE TABLE test_tab_2");
> +$node_publisher->wait_for_catchup($appname);
> $node_subscriber->safe_psql('postgres',
> - "ALTER SUBSCRIPTION tap_sub SET (binary = on)");
> + "CREATE UNIQUE INDEX idx_tab on test_tab_2(a)");
>
> What is the need to first Drop the index and then recreate it after a few lines?

Since we want the two transactions to complete normally without conflicts due
to the unique index, we temporarily delete the index.
I added some new comments to explain this.

> 4. Attached, find some comment improvements atop v67-0002* patch.
> Similar comments need to be changed in other test files.

Thanks, I have checked and merge them. And also changed similar comments in
other test files.

> 5. Attached, find some comment improvements atop v67-0001* patch.

Thanks, I have checked and merge them.

Attach the new version patch which addressed all above comments and part of
comments from [1] except one comment that are being discussed.

[1] - https://www.postgresql.org/message-id/CAD21AoDvT%2BTv3auBBShk19EkKLj6ByQtnAzfMjh49BhyT7f4Nw%40mail.gmail.com

Regards,
Wang wei

Attachment Content-Type Size
v68-0001-Perform-streaming-logical-transactions-by-parall.patch application/octet-stream 214.8 KB
v68-0002-Test-streaming-parallel-option-in-tap-test.patch application/octet-stream 62.2 KB
v68-0003-Add-GUC-stream_serialize_threshold-and-test-seri.patch application/octet-stream 12.4 KB
v68-0004-Stop-extra-worker-if-GUC-was-changed.patch application/octet-stream 4.6 KB
v68-0005-Retry-to-apply-streaming-xact-only-in-apply-work.patch application/octet-stream 21.2 KB
v68-0006-Add-a-main_worker_pid-to-pg_stat_subscription.patch application/octet-stream 9.5 KB

In response to

Responses

Browse pgsql-hackers by date

  From Date Subject
Next Message wangw.fnst@fujitsu.com 2022-12-27 04:57:00 RE: Perform streaming logical transactions by background workers and parallel apply
Previous Message Amit Kapila 2022-12-27 04:02:52 Re: Time delayed LR (WAS Re: logical replication restrictions)