RE: Failed transaction statistics to measure the logical replication progress

On Tues, Sep 28, 2021 6:05 PM Amit Kapila <amit(dot)kapila16(at)gmail(dot)com> wrote:
> On Tue, Sep 28, 2021 at 11:35 AM Masahiko Sawada <sawada(dot)mshk(at)gmail(dot)com> wrote:
> >
> > On Tue, Sep 28, 2021 at 1:54 PM Amit Kapila <amit(dot)kapila16(at)gmail(dot)com>
> wrote:
> > >
> > > >
> > > > Then, if, we proceed in this direction,
> > > > the place to implement those stats
> > > > would be on the LogicalRepWorker struct, instead ?
> > > >
> > >
> > > Or, we can make existing stats persistent and then add these stats on
> > > top of it. Sawada-San, do you have any thoughts on this matter?
> >
> > I think that making existing stats including received_lsn and
> > last_msg_receipt_time persistent by using stats collector could cause
> > massive reporting messages. We can report these messages with a
> > certain interval to reduce the amount of messages but we will end up
> > seeing old stats on the view.
> >
>
> Can't we keep the current and new stats both in-memory and persist on
> disk? So, the persistent stats data will be used to fill the in-memory
> counters after restarting of workers, otherwise, we will always refer
> to in-memory values.

I think this approach works, but I have another concern about it.

The current pg_stat_subscription view is listed as "Dynamic Statistics Views" in
the document, the data in it seems about the worker process, and the view data
shows what the current worker did. But if we keep the new xact stat persist,
then it's not what the current worker did, it looks more related to the
subscription historic data.

Adding a new view seems resonalble, but it will bring another subscription
related view which might be too much. OTOH, I can see there are already some
different views[1] including xact stat, maybe adding another one is accepatble ?

[1]
pg_stat_xact_all_tables
pg_stat_xact_sys_tables
pg_stat_xact_user_tables
pg_stat_xact_user_functions

Best regards,
Hou zj