Re: Memory consumed by paths during partitionwise join planning

On Thu, Feb 15, 2024 at 9:41 AM Andrei Lepikhov
<a(dot)lepikhov(at)postgrespro(dot)ru> wrote:
>
> On 6/2/2024 19:51, Ashutosh Bapat wrote:
> > On Fri, Dec 15, 2023 at 5:22 AM Ashutosh Bapat
> > The patches are raw. make check has some crashes that I need to fix. I
> > am waiting to hear whether this is useful and whether the design is on
> > the right track.
> Let me write words of opinion on that feature.
> I generally like the idea of a refcount field. We had problems
> generating alternative paths in extensions and designing the Asymmetric
> Join feature [1] when we proposed an alternative path one level below
> and called the add_path() routine. We had lost the path in the path list
> referenced from the upper RelOptInfo. This approach allows us to make
> more handy solutions instead of hacking with a copy/restore pathlist.

Thanks for your comments. I am glad that you find this useful.

> But I'm not sure about freeing unreferenced paths. I would have to see
> alternatives in the pathlist.

I didn't understand this. Can you please elaborate? A path in any
pathlist is referenced. An unreferenced path should not be in any
pathlist.

> About partitioning. As I discovered planning issues connected to
> partitions, the painful problem is a rule, according to which we are
> trying to use all nomenclature of possible paths for each partition.
> With indexes, it quickly increases optimization work. IMO, this can help
> a 'symmetrical' approach, which could restrict the scope of possible
> pathways for upcoming partitions if we filter some paths in a set of
> previously planned partitions.

filter or free?

> Also, I am glad to see a positive opinion about the path_walker()
> routine. Somewhere else, for example, in [2], it seems we need it too.
>

I agree that we should introduce path_walker() yes.

I am waiting for David's opinion about the performance numbers shared
in [1] before working further on the patches and bring them out of WIP
state.

[1] postgr.es/m/CAExHW5uDkMQL8SicV3_=AYcsWwMTNR8GzJo310J7sv7TMLoL6Q@mail.gmail.com

--
Best Wishes,
Ashutosh Bapat