RE: [PATCH] Reuse Workers and Replication Slots during Logical Replication

From: "Hayato Kuroda (Fujitsu)" <kuroda(dot)hayato(at)fujitsu(dot)com>
To: 'Melih Mutlu' <m(dot)melihmutlu(at)gmail(dot)com>
Cc: Peter Smith <smithpb2250(at)gmail(dot)com>, Melanie Plageman <melanieplageman(at)gmail(dot)com>, "Wei Wang (Fujitsu)" <wangw(dot)fnst(at)fujitsu(dot)com>, "Yu Shi (Fujitsu)" <shiy(dot)fnst(at)fujitsu(dot)com>, Amit Kapila <amit(dot)kapila16(at)gmail(dot)com>, pgsql-hackers <pgsql-hackers(at)postgresql(dot)org>, shveta malik <shveta(dot)malik(at)gmail(dot)com>
Subject: RE: [PATCH] Reuse Workers and Replication Slots during Logical Replication
Date: 2023-06-27 07:42:49
Message-ID: TYAPR01MB5866144D41E44697979A0CC0F527A@TYAPR01MB5866.jpnprd01.prod.outlook.com
Views: Raw Message | Whole Thread | Download mbox | Resend email
Thread:
Lists: pgsql-hackers

Dear Melih,

Thank you for updating the patch! I have not reviewed yet, but I wanted
to reply your comments.

> This actually makes sense. I quickly try to do that without adding any
> new replication message. As you would expect, it did not work.
> I don't really know what's needed to make a connection to last for
> more than one iteration. Need to look into this. Happy to hear any
> suggestions and thoughts.

I have analyzed how we handle this. Please see attached the patch (0003) which
allows reusing connection. The patchset passed tests on my CI.
To make cfbot happy I reassigned the patch number.

In this patch, the tablesync worker does not call clean_sync_worker() at the end
of iterations, and the establishment of the connection is done only once.
The creation of memory context is also suppressed.

Regarding the walsender, streamingDone{Sending|Receiving} is now initialized
before executing StartLogicalReplication(). These flags have been used to decide
when the process exits copy mode. The default value is false, and they are set
to true when the copy mode is finished.
I think there was no use-case that the same walsender executes START_REPLICATION
replication twice so there were no codes for restoring flags. Please tell me if any other
reasons.

Best Regards,
Hayato Kuroda
FUJITSU LIMITED

Attachment Content-Type Size
0004-Add-replication-protocol-cmd-to-create-a-snapshot.patch application/octet-stream 21.0 KB
0005-Reuse-Replication-Slot-and-Origin-in-Tablesync.patch application/octet-stream 55.8 KB
0001-Refactor-to-split-Apply-and-Tablesync-Workers.patch application/octet-stream 21.1 KB
0002-Reuse-Tablesync-Workers.patch application/octet-stream 10.4 KB
0003-reuse-connection-when-tablesync-workers-change-the-t.patch application/octet-stream 3.8 KB

In response to

Responses

Browse pgsql-hackers by date

  From Date Subject
Next Message Yugo NAGATA 2023-06-27 07:48:45 Re: pgbnech: allow to cancel queries during benchmark
Previous Message Kyotaro Horiguchi 2023-06-27 07:41:00 Re: Add TLI number to name of files generated by pg_waldump --save-fullpage