Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions

On Tue, Dec 10, 2019 at 9:52 AM Amit Kapila <amit(dot)kapila16(at)gmail(dot)com> wrote:
>
> On Mon, Dec 2, 2019 at 2:02 PM Dilip Kumar <dilipbalaut(at)gmail(dot)com> wrote:
> >
> > On Sun, Dec 1, 2019 at 7:58 AM Michael Paquier <michael(at)paquier(dot)xyz> wrote:
> > >
> > > On Fri, Nov 22, 2019 at 01:18:11PM +0530, Dilip Kumar wrote:
> > > > I have rebased the patch on the latest head and also fix the issue of
> > > > "concurrent abort handling of the (sub)transaction." and attached as
> > > > (v1-0013-Extend-handling-of-concurrent-aborts-for-streamin) along with
> > > > the complete patch set. I have added the version number so that we
> > > > can track the changes.
> > >
> > > The patch has rotten a bit and does not apply anymore. Could you
> > > please send a rebased version? I have moved it to next CF, waiting on
> > > author.
> >
> > I have rebased the patch set on the latest head.
> >
> > Apart from this, there is one issue reported by my colleague Vignesh.
> > The issue is that if we use more than two relations in a transaction
> > then there is an error on standby (no relation map entry for remote
> > relation ID 16390). After analyzing I have found that for the
> > streaming transaction an "is_schema_sent" flag is kept in
> > ReorderBufferTXN. And, I think that is done so that we can send the
> > schema for each transaction stream so that if any subtransaction gets
> > aborted we don't lose the logical WAL for that schema. But, this
> > solution has induced a very basic issue that if a transaction operate
> > on more than 1 relation then after sending the schema for the first
> > relation it will mark the flag true and the schema for the subsequent
> > relations will never be sent.
> >
>
> How about keeping a list of top-level xids in each RelationSyncEntry?
> Basically, whenever we send the schema for any transaction, we note
> that in RelationSyncEntry and at abort time we can remove xid from the
> list. Now, whenever, we check whether to send schema for any
> operation in a transaction, we will check if our xid is present in
> that list for a particular RelationSyncEntry and take an action based
> on that (if xid is present, then we won't send the schema, otherwise,
> send it).
The idea make sense to me. I will try to write a patch for this and test.

--
Regards,
Dilip Kumar
EnterpriseDB: http://www.enterprisedb.com