Re: Skipping logical replication transactions on subscriber side

On Tue, Dec 14, 2021 at 1:07 PM Dilip Kumar <dilipbalaut(at)gmail(dot)com> wrote:
>
> On Tue, Dec 14, 2021 at 8:20 AM Amit Kapila <amit(dot)kapila16(at)gmail(dot)com> wrote:
> >
> > On Mon, Dec 13, 2021 at 6:55 PM Masahiko Sawada <sawada(dot)mshk(at)gmail(dot)com> wrote:
>
> > > In streaming cases, we don’t know when stream-commit or stream-abort
> > > comes and another conflict could occur on the subscription in the
> > > meanwhile. But given that (we expect) this feature is used after the
> > > apply worker enters into an error loop, this is unlikely to happen in
> > > practice unless the user sets the wrong XID. Similarly, in 2PC cases,
> > > we don’t know when commit-prepared or rollback-prepared comes and
> > > another conflict could occur in the meanwhile. But this could occur in
> > > practice even if the user specified the correct XID. Therefore, if we
> > > disallow to change skip_xid until the subscriber receives
> > > commit-prepared or rollback-prepared, we cannot skip the second
> > > transaction that conflicts with data on the subscriber.
> > >
> >
> > I agree with this theory. Can we reflect this in comments so that in
> > the future we know why we didn't pursue this direction?
>
> I might be missing something here, but for streaming, transaction
> users can decide whether they wants to skip or not only once we start
> applying no? I mean only once we start applying the changes we can
> get some errors and by that time we must be having all the changes for
> the transaction.
>

That is right and as per my understanding, the patch is trying to
accomplish the same.

> So I do not understand the point we are trying to
> discuss here?
>

The point is that whether we can skip the changes while streaming
itself like when we get the changes and write to a stream file. Now,
it is possible that streams from multiple transactions can be
interleaved and users can change the skip_xid in between. It is not
that we can't handle this but that would require a more complex design
and it doesn't seem worth it because we can anyway skip the changes
while applying as you mentioned in the previous paragraph.

--
With Regards,
Amit Kapila.