RE: Failed transaction statistics to measure the logical replication progress

On Thursday, September 30, 2021 11:53 AM Hou, Zhijie/侯 志杰 <houzj(dot)fnst(at)fujitsu(dot)com> wrote:
> On Tues, Sep 28, 2021 6:05 PM Amit Kapila <amit(dot)kapila16(at)gmail(dot)com> wrote:
> > On Tue, Sep 28, 2021 at 11:35 AM Masahiko Sawada
> <sawada(dot)mshk(at)gmail(dot)com> wrote:
> > >
> > > On Tue, Sep 28, 2021 at 1:54 PM Amit Kapila
> > > <amit(dot)kapila16(at)gmail(dot)com>
> > wrote:
> > > >
> > > > >
> > > > > Then, if, we proceed in this direction, the place to implement
> > > > > those stats would be on the LogicalRepWorker struct, instead ?
> > > > >
> > > >
> > > > Or, we can make existing stats persistent and then add these stats
> > > > on top of it. Sawada-San, do you have any thoughts on this matter?
> > >
> > > I think that making existing stats including received_lsn and
> > > last_msg_receipt_time persistent by using stats collector could
> > > cause massive reporting messages. We can report these messages with
> > > a certain interval to reduce the amount of messages but we will end
> > > up seeing old stats on the view.
> > >
> >
> > Can't we keep the current and new stats both in-memory and persist on
> > disk? So, the persistent stats data will be used to fill the in-memory
> > counters after restarting of workers, otherwise, we will always refer
> > to in-memory values.
>
> I think this approach works, but I have another concern about it.
>
> The current pg_stat_subscription view is listed as "Dynamic Statistics Views"
> in the document, the data in it seems about the worker process, and the view
> data shows what the current worker did. But if we keep the new xact stat
> persist, then it's not what the current worker did, it looks more related to the
> subscription historic data.
>
> Adding a new view seems resonalble, but it will bring another subscription
> related view which might be too much. OTOH, I can see there are already some
> different views[1] including xact stat, maybe adding another one is
> accepatble ?
I think we'll try to suppress the increment of the numbers
of subscription related stats, if the possibility is not denied.

In terms of the document you mentioned,
I feel I'd need some modifications to it in the patch,
based on the change.

Best Regards,
Takamichi Osumi