Re: segfault with incremental sort

On Mon, Nov 23, 2020 at 7:23 PM James Coleman <jtc331(at)gmail(dot)com> wrote:
>
> On Mon, Nov 23, 2020 at 7:56 AM Amit Kapila <amit(dot)kapila16(at)gmail(dot)com> wrote:
> >
> > On Sat, Nov 21, 2020 at 1:21 AM James Coleman <jtc331(at)gmail(dot)com> wrote:
> >
> > IIRC, the reason was that for correlated subplans each time we need to
> > send the param for execution to workers, and for that, we don't have
> > an implementation yet. Basically, if the param size changes each time
> > (say for varchar type of params), the amount of memory required would
> > be different each time. It is not that we can't implement it but I
> > think we have left it originally because we were not sure of the
> > number of cases it can benefit and certainly it needs some more work.
> > I am not following this and related discussions closely but can
> > explain to me why you think the query/plan you are talking about is
> > safe with respect to the above hazard?
>
> Thanks for the explanation.
>
> In this particular case we're not dealing with variable length fields
> (it's an int), so that particular problem wouldn't inherently apply
> (though I understand the generalized rule).
>
> But I'm not really quite sure how sending params to workers (from the
> leader I assume) is relevant here. In another thread you can see the
> full plan [1], but effectively we have:
>
> Gather Merge
> Sort
> Nested Loop
> Bitmap Heap Scan
> Index Only Scan
> Subplan 1
> Subplan 2
>
> where the two subplans are returning the single result of a correlated
> subquery (in a SELECT). As I understand it this doesn't require any
> coordination with/from the leader at all; all of the information in
> this case should actually be local to the individual worker. Each
> worker needs to, for each tuple in its index scan, do another index
> lookup based on that tuple.
>

Yeah, this doesn't require any communication between leader and worker
but to enable it for such cases, we need to identify when we have
correlated subquery where we can make it parallel-safe and then
probably allow it. IIRC, we only allow cases of un-correlated subplans
and initplans when those are at the same or a level above the Gather
(Merge) node. Now, I think it is quite possible that in PG-13 we have
unintentionally allowed plans containing correlated sub-queries but
missed to take care of it at all required places. So, that could be
the reason why we are not seeing any such problems before PG-13? To
prove/disprove this theory, we need to see if we can produce a similar
problem in PG-12 where we don't have incremental_sort or maybe in
PG-13 by disabling incremental_sort. If we have introduced it with
incremental_sort, then we need to either ensure that such plans are
not parallel-safe as would be the case before PG-13 or see what else
needs to be done to support in all such possible cases.

--
With Regards,
Amit Kapila.