Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions

On Sun, Mar 29, 2020 at 11:19:21AM +0530, Amit Kapila wrote:
>On Sun, Mar 29, 2020 at 6:29 AM Tomas Vondra
><tomas(dot)vondra(at)2ndquadrant(dot)com> wrote:
>>
>> On Sat, Mar 28, 2020 at 03:29:34PM +0530, Amit Kapila wrote:
>> >On Sat, Mar 28, 2020 at 2:19 PM Dilip Kumar <dilipbalaut(at)gmail(dot)com> wrote:
>> >
>> >How about if instead of writing an XLOG_XACT_ASSIGNMENT WAL, we set a
>> >flag in TransactionStateData and then log that as special information
>> >whenever we write next WAL record for a new subtransaction? Then
>> >during recovery, we can only call ProcArrayApplyXidAssignment when we
>> >find that special flag is set in a WAL record. One idea could be to
>> >use a flag bit in XLogRecord.xl_info. If that is feasible then the
>> >solution can work as it is now, without any overhead or change in the
>> >way we maintain KnownAssignedXids.
>> >
>>
>> Ummm, how is that different from what the patch is doing now? I mean, we
>> only write the top-level XID for the first WAL record in each subxact,
>> right? Or what would be the difference with your approach?
>>
>
>We have to do what the patch is currently doing and additionally, we
>will set this flag after PGPROC_MAX_CACHED_SUBXIDS which would allow
>us to call ProcArrayApplyXidAssignment during WAL replay only after
>PGPROC_MAX_CACHED_SUBXIDS number of subxacts. It will help us in
>clearing the KnownAssignedXids at the same time as we do now, so no
>additional performance overhead.
>

Hmmm. So we'd still log assignment twice? Or would we keep just the
immediate assignments (embedded into xlog records), and cache the
subxids on the replica somehow?

regards

--
Tomas Vondra http://www.2ndQuadrant.com
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services