Re: Buffers from parallel workers not accumulated to upper nodes with gather merge

From: Jehan-Guillaume de Rorthais <jgdr(at)dalibo(dot)com>
To: Amit Kapila <amit(dot)kapila16(at)gmail(dot)com>
Cc: PostgreSQL mailing lists <pgsql-bugs(at)lists(dot)postgresql(dot)org>
Subject: Re: Buffers from parallel workers not accumulated to upper nodes with gather merge
Date: 2020-07-23 09:07:49
Message-ID: 20200723110749.7aa0e15a@firost
Views: Raw Message | Whole Thread | Download mbox | Resend email
Thread:
Lists: pgsql-bugs

On Tue, 21 Jul 2020 17:34:57 +0200
Jehan-Guillaume de Rorthais <jgdr(at)dalibo(dot)com> wrote:
> On Mon, 20 Jul 2020 15:29:00 +0530
> Amit Kapila <amit(dot)kapila16(at)gmail(dot)com> wrote:
> > On Mon, Jul 20, 2020 at 1:29 PM Jehan-Guillaume de Rorthais
> > <jgdr(at)dalibo(dot)com> wrote:
> > > On Mon, 20 Jul 2020 11:30:34 +0530
> > > Amit Kapila <amit(dot)kapila16(at)gmail(dot)com> wrote:
[...]
> > > > I think irrespective of whether we want to call
> > > > ExecShutdownGatherMergeWorkers in gather_merge_getnext (when we know
> > > > we are done aka binaryheap_empty) to fix this particular issue, it is
> > > > better to shutdown the workers as soon as we are done similar to what
> > > > we do for Gather node. It is good to release resources as soon as we
> > > > can.
> > >
> > > Indeed. I was focusing on the bug and I didn't thought about that. The
> > > patch I test was really just a quick test. I'll have a closer look at
> > > this, but I suppose this might be considered as a separate commit?
> > >
> >
> > Good Question. Initially, I thought we will have it in a same commit,
> > but now on again thinking about it, we might want to keep this one for
> > the HEAD only and ExecShutdownNode related change in the back-branches.
>
> OK.
>
> I'll send a patch proposal for Gather Merge on HEAD tomorrow.

Please, find in attachment the patch to release Gather Merge workers as soon as
they are exhausted.

I was wondering how interesting it would be to release each worker individually
of each others? This might be interesting for long running queries and/or large
number of workers. This would apply to Gather as well. Do you think it worth to
give it try? Maybe it has already been discussed in the past?

Regards,

Attachment Content-Type Size
v1-0001-Shutdown-Gather-Merge-workers-when-exhausted.patch text/x-patch 904 bytes

In response to

Responses

Browse pgsql-bugs by date

  From Date Subject
Next Message Amit Kapila 2020-07-23 10:14:16 Re: Buffers from parallel workers not accumulated to upper nodes with gather merge
Previous Message Slawomir Nowakiewicz 2020-07-23 06:49:30 RE: BUG #16549: "CASE" not work properly , the function works properly on PostgreSQL 9.6.8