Re: Control flow in logical replication walsender

On Tue, Apr 30, 2024 at 11:28 PM Christophe Pettus <xof(at)thebuild(dot)com> wrote:

>
> Hi,
>
> I wanted to check my understanding of how control flows in a walsender
> doing logical replication. My understanding is that the (single) thread in
> each walsender process, in the simplest case, loops on:
>
> 1. Pull a record out of the WAL.
> 2. Pass it to the recorder buffer code, which,
> 3. Sorts it out into the appropriate in-memory structure for that
> transaction (spilling to disk as required), and then continues with #1, or,
> 4. If it's a commit record, it iteratively passes the transaction data one
> change at a time to,
> 5. The logical decoding plugin, which returns the output format of that
> plugin, and then,
> 6. The walsender sends the output from the plugin to the client. It cycles
> on passing the data to the plugin and sending it to the client until it
> runs out of changes in that transaction, and then resumes reading the WAL
> in #1.
>
>
This is correct barring some details on master.

> In particular, I wanted to confirm that while it is pulling the reordered
> transaction and sending it to the plugin (and thence to the client), that
> particular walsender is *not* reading new WAL records or putting them in
> the reorder buffer.
>
>
This is correct.

> The specific issue I'm trying to track down is an enormous pileup of spill
> files. This is in a non-supported version of PostgreSQL (v11), so an
> upgrade may fix it, but at the moment, I'm trying to find a cause and a
> mitigation.
>
>
Is there a large transaction which is failing to be replicated repeatedly -
timeouts, crashes on upstream or downstream?

--
Best Wishes,
Ashutosh Bapat