Re: Perform streaming logical transactions by background workers and parallel apply

On Fri, Feb 3, 2023 at 12:29 PM houzj(dot)fnst(at)fujitsu(dot)com
<houzj(dot)fnst(at)fujitsu(dot)com> wrote:
>
> On Friday, February 3, 2023 11:04 AM Amit Kapila <amit(dot)kapila16(at)gmail(dot)com> wrote:
> >
> > On Thu, Feb 2, 2023 at 4:52 AM Peter Smith <smithpb2250(at)gmail(dot)com>
> > wrote:
> > >
> > > Some minor review comments for v91-0001
> > >
> >
> > Pushed this yesterday after addressing your comments!
>
> Thanks for pushing.
>
> Currently, we have two remaining patches which we are not sure whether it's worth
> committing for now. Just share them here for reference.
>
> 0001:
>
> Based on our discussion[1] on -hackers, it's not clear that if it's necessary
> to add the sub-feature to stop extra worker when
> max_apply_workers_per_suibscription is reduced. Because:
>
> - it's not clear whether reducing the 'max_apply_workers_per_suibscription' is very
> common.

A use case I'm concerned about is a temporarily intensive data load,
for example, a data loading batch job in a maintenance window. In this
case, the user might want to temporarily increase
max_parallel_workers_per_subscription in order to avoid a large
replication lag, and revert the change back to normal after the job.
If it's unlikely to stream the changes in the regular workload as
logical_decoding_work_mem is big enough to handle the regular
transaction data, the excess parallel workers won't exit. Another
subscription might want to use parallel workers but there might not be
free workers. That's why I thought we need to free the excess workers
at some point.

Regards,

--
Masahiko Sawada
Amazon Web Services: https://aws.amazon.com