Re: Skipping logical replication transactions on subscriber side

On Fri, Oct 8, 2021 at 4:09 PM osumi(dot)takamichi(at)fujitsu(dot)com
<osumi(dot)takamichi(at)fujitsu(dot)com> wrote:
>
> On Thursday, September 30, 2021 2:45 PM Masahiko Sawada <sawada(dot)mshk(at)gmail(dot)com> wrote:
> > I've attached updated patches that incorporate all comments I got so far. Please
> > review them.
> Hi
>
>
> Sorry, if I misunderstand something but
> did someone check what happens when we
> execute ALTER SUBSCRIPTION ... RESET (streaming)
> in the middle of one txn which has several streaming of data to the sub,
> especially after some part of txn has been already streamed.
> My intention of this is something like *if* we can find an actual harm of this,
> I wanted to suggest the necessity of a safeguard or some measure into the patch.
>
> An example)
>
> Set the logical_decoding_work_mem = 64kB on the pub.
> and create a table and subscription with streaming = true.
> In addition, log_min_messages = DEBUG1 on the sub
> is helpful to check the LOG on the sub in stream_open_file().
>
> <Session 1> connect to the publisher
>
> BEGIN;
> INSERT INTO tab VALUES (generate_series(1, 1000)); -- this exceeds the memory limit
> SELECT * FROM pg_stat_replication_slots; -- check the actual streaming bytes&counts just in case
>
> <Session 2> connect to the subscriber
> -- after checking some logs of "open file .... for streamed changes" on the sub
> ALTER SUBSCRIPTION mysub RESET (streaming)
>
> <Session 1>
> INSERT INTO tab VALUES (generate_series(1001, 2000)); -- again, exceeds the limit
> COMMIT;
>
>
> I observed that the subscriber doesn't
> accept STREAM_COMMIT in this case but gets BEGIN&COMMIT instead at the end.
> I couldn't find any apparent and immediate issue from those steps
> but is that no problem ?
> Probably, this kind of situation applies to other reset target options ?

I think that if a subscription parameter such as ‘streaming’ and
‘binary’ is changed, an apply worker exits and the launcher starts a
new worker (see maybe_reread_subscription()). So I guess that in this
case, the apply worker exited during receiving streamed changes,
restarted, and received the same changes with ‘streaming = off’,
therefore it got BEGIN and COMMIT instead. I think that this happens
even by using ‘SET (‘streaming’ = off)’.

Regards,

--
Masahiko Sawada
EDB: https://www.enterprisedb.com/