Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions

On Sat, Mar 28, 2020 at 2:19 PM Dilip Kumar <dilipbalaut(at)gmail(dot)com> wrote:
>
> On Sat, Mar 28, 2020 at 11:56 AM Amit Kapila <amit(dot)kapila16(at)gmail(dot)com> wrote:
> >
> >
> > I have looked at the solution proposed and I would like to share my
> > findings. I think calling ProcArrayApplyXidAssignment for each
> > subtransaction is not a good idea for a couple of reasons:
> > (a) It will just beat the purpose of maintaining KnowAssignedXids
> > array which is to avoid looking at pg_subtrans in
> > TransactionIdIsInProgress() on standby. Basically, if we remove it
> > for each subXid, it will consider the KnowAssignedXids to be
> > overflowed and check pg_subtrans frequently.
>
> Right, I also think this is a problem with this solution. I think we
> may try to avoid this by caching this information. But, then we will
> have to maintain this in some dimensional array which stores
> sub-transaction ids per top transaction or we can maintain a list of
> sub-transaction for each transaction. I haven't thought about how
> much complexity this solution will add.
>

How about if instead of writing an XLOG_XACT_ASSIGNMENT WAL, we set a
flag in TransactionStateData and then log that as special information
whenever we write next WAL record for a new subtransaction? Then
during recovery, we can only call ProcArrayApplyXidAssignment when we
find that special flag is set in a WAL record. One idea could be to
use a flag bit in XLogRecord.xl_info. If that is feasible then the
solution can work as it is now, without any overhead or change in the
way we maintain KnownAssignedXids.

--
With Regards,
Amit Kapila.
EnterpriseDB: http://www.enterprisedb.com