Re: Skipping logical replication transactions on subscriber side

On Thu, Oct 28, 2021 at 1:05 PM Amit Kapila <amit(dot)kapila16(at)gmail(dot)com> wrote:
>
> On Thu, Oct 28, 2021 at 7:49 AM Masahiko Sawada <sawada(dot)mshk(at)gmail(dot)com> wrote:
> >
> > On Wed, Oct 27, 2021 at 2:43 PM Amit Kapila <amit(dot)kapila16(at)gmail(dot)com> wrote:
> > >
> > > On Wed, Oct 27, 2021 at 10:43 AM Masahiko Sawada <sawada(dot)mshk(at)gmail(dot)com> wrote:
> > > >
> > > > > > BTW how useful is specifying LSN instead of XID in practice? Given
> > > > > > that this skipping behavior is used to skip the particular transaction
> > > > > > (or its part of operations) in question, I’m not sure specifying LSN
> > > > > > or time is useful.
> > > > > >
> > > > >
> > > > > I think if the user wants to skip multiple xacts, she might want to
> > > > > use the highest LSN to skip instead of specifying individual xids.
> > > >
> > > > I think it assumes that the situation where the user already knows
> > > > multiple transactions that cannot be applied on the subscription but
> > > > how do they know?
> > > >
> > >
> > > Either from the error messages in the server log or from the new view
> > > we are planning to add. I think such a case is possible during the
> > > initial synchronization phase where apply worker went ahead then
> > > tablesync worker by skipping to apply the changes on the corresponding
> > > table. After that it is possible, that table sync worker failed during
> > > copy and apply worker fails during the processing of some other rel.
> >
> > Does it mean that if both initial copy for the corresponding table by
> > table sync worker and applying changes for other rels by apply worker
> > fail, we skip both by specifying LSN?
> >
>
> Yes.
>
> > If so, can't we disable the
> > initial copy for the table and skip only the changes for other rels
> > that cannot be applied?
> >
>
> But anyway you need some way to skip changes via a particular
> tablesync worker so that it can mark the relation in 'ready' state.

Right.

> I
> think one can also try to use disable_on_error option in such
> scenarios depending on how we expose it. Say, if the option means that
> all workers (apply or table sync) should be disabled on an error then
> it would be a bit tricky but if we can come up with a way to behave
> differently for different workers then it is possible to disable one
> set of workers and skip the changes in another set of workers.

Yes, I would prefer to skip individual transactions in question rather
than skip changes until the particular LSN. It’s not advisable to use
LSN to skip changes since it has a risk of skipping unrelated changes
too.

Regards,

--
Masahiko Sawada
EDB: https://www.enterprisedb.com/