Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions

On Wed, Apr 8, 2020 at 6:29 AM Tomas Vondra
<tomas(dot)vondra(at)2ndquadrant(dot)com> wrote:
>
> On Tue, Apr 07, 2020 at 12:17:44PM +0530, Amit Kapila wrote:
> >On Mon, Mar 30, 2020 at 8:58 PM Tomas Vondra
> ><tomas(dot)vondra(at)2ndquadrant(dot)com> wrote:
> >>
> >> On Mon, Mar 30, 2020 at 11:47:57AM +0530, Amit Kapila wrote:
> >> >
> >> >I think we need to cache the subxids on the replica somehow but I
> >> >don't have a very good idea for it. Basically, there are two ways to
> >> >do it (a) Change the KnownAssignedXids in some way so that we can
> >> >easily find this information without losing on the current benefits of
> >> >it. I can't think of a good way to do that and even if we come up
> >> >with something, it could easily be a lot of work, (b) Cache the
> >> >subxids for a particular transaction in local memory along with
> >> >KnownAssignedXids. This is doable but now we have two data-structures
> >> >(one in shared memory and other in local memory) managing the same
> >> >information in different ways.
> >> >
> >> >Do you have any other ideas?
> >>
> >> I don't follow. Why couldn't we have a simple cache on the standby? It
> >> could be either a simple array or a hash table (with the top-level xid
> >> as hash key)?
> >>
> >
> >I think having something like we discussed or what you have in the
> >patch won't be sufficient to clean the KnownAssignedXid array. The
> >point is that we won't write a WAL for xid-subxid association for
> >unlogged relations in the "Immediately WAL-log assignments" patch,
> >however, the KnownAssignedXid would have both kinds of Xids as we
> >autofill it with gaps (see RecordKnownAssignedTransactionIds). I
> >think if my understanding is correct to make it work we might need
> >major surgery in the code or have to maintain KnownAssignedXid array
> >differently.
>
> Hmm, that's a good point. If I understand correctly, the issue is
> that if we create new subxact, write something into an unlogged table,
> and then create new subxact, the XID of the first subxact will be "known
> assigned" but we won't know it's a subxact or to which parent xact it
> belongs (because there will be no WAL records that could encode it).
>
> I wonder if there's a simple solution (e.g. when creating the second
> subxact we might notice the xid-subxid assignment was not logged, and
> write some "dummy" WAL record). But I admit it seems a bit ugly.
>
> >>
> >> I don't think this is particularly complicated or a lot of code, and I
> >> don't see why would it require data structures in shared memory. Only
> >> the walreceiver on standby needs to worry about this, no?
> >>
> >
> >Not a new data structure in shared memory, but we already have a
> >KnownTransactionId structure in shared memory. So, after having a
> >local cache, we will have xidAssignmentsHash and KnownTransactionId
> >maintaining the same information in different ways. And, we need to
> >ensure both are cleaned up properly. That was what I was pointing
> >above related to maintaining two structures. However, I think before
> >discussing more on this, we need to think about the above problem.

I have rebased the patch on the latest head. I haven't yet changed
anything for xid assignment thing because it is not yet concluded.

--
Regards,
Dilip Kumar
EnterpriseDB: http://www.enterprisedb.com