Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions

On Wed, Nov 13, 2019 at 5:55 PM Amit Kapila <amit(dot)kapila16(at)gmail(dot)com> wrote:
>
> On Thu, Oct 3, 2019 at 1:18 PM Dilip Kumar <dilipbalaut(at)gmail(dot)com> wrote:
> >
>
> As mentioned by me a few days back that the first patch in this series
> is ready to go [1] (I am hoping Tomas will pick it up), so I have
> started the review of other patches
>
> Review/Questions on 0002-Immediately-WAL-log-assignments.patch
> -------------------------------------------------------------------------------------------------
> 1. This patch adds the top_xid in WAL whenever the first time WAL for
> a subtransaction XID is written to correctly decode the changes of
> in-progress transaction. This patch also removes logging and applying
> WAL for XLOG_XACT_ASSIGNMENT which might have some effect. As replay
> of that, it prunes KnownAssignedXids to prevent overflow of that
> array. See comments in procarray.c (KnownAssignedTransactionIds
> sub-module). Can you please explain how after removing the WAL for
> XLOG_XACT_ASSIGNMENT, we will handle that or I am missing something
> and there is no impact of same?

It seems like a problem to me as well. One option could be that
since now we are adding the top transaction id in the first WAL of the
subtransaction we can directly update the pg_subtrans and avoid adding
sub transaction id in the KnownAssignedXids and mark it as
lastOverflowedXid. But, I don't think we should go in that direction
otherwise it will impact the performance of visibility check on the
hot-standby. Let's see what Tomas has in mind.

>
> 2.
> +#define XLOG_INCLUDE_INVALS 0x08 /* include invalidations */
>
> This doesn't seem to be used in this patch.

>
--
Regards,
Dilip Kumar
EnterpriseDB: http://www.enterprisedb.com