Re: Add mention of execution time memory for enable_partitionwise_* GUCs

On Fri, 19 Jul 2024 at 17:24, Ashutosh Bapat
<ashutosh(dot)bapat(dot)oss(at)gmail(dot)com> wrote:
> According to [1] these GUCs were added because of increased
> memory consumption during planning and time taken to plan the query.
> The execution time memory consumption issue was known even back then
> but the GUC was not set to default because of that. But your patch
> proposes to change the narrative. In the same thread [1], you will
> find the discussion about turning the default to ON once we have fixed
> planner's memory and time consumption. We have patches addressing
> those issues [2] [3]. With your proposed changes we will almost never
> have a chance to turn those GUCs ON by default. That seems a rather
> sad prospect.

Sad prospect for who? If the day comes and we're considering enabling
these GUCs by default, I think we'll likely also be considering if
it'll be sad for users who get an increased likelihood of OOM kills
because the chosen plan uses $nparts times more memory to execute than
the old plan.

Can you honestly say that you have no concern about increased executor
memory consumption if we switched on these GUCs by default? I was
certainly concerned about it in [5], so I dropped that patch after
realising what could happen.

> I am fine if we want to mention that the executor may consume a large
> amount of memory when these GUCs are turned ON. Users may decide to
> turn those OFF if they can not afford to spend that much memory during
> execution. But I don't like tying execution time consumption with
> default OFF.

If you were to fix the planner issues then this text would need to be
revisited anyway. However, I seriously question your judgment on
fixing those alone being enough to allow us to switch these on by
default. It seems unlikely that the planner would use anything near
any realistic work_mem setting extra memory per partition, but that's
what the executor would do, given enough data per partition.

I'd say any analysis that only found planner memory and time to be the
only reason to turn these GUCs off by default was incomplete analysis.
Maybe it was a case of stopping looking for all the reasons once
enough had been found to make the choice. If not, then they found the
molehill and failed to notice the mountain.

David

[4] https://postgr.es/m/3603c380-d094-136e-e333-610914fb3e80%40gmx.net
[5] https://postgr.es/m/CAApHDvojKdBR3MR59JXmaCYbyHB6Q_5qPRU+dy93En8wm+XiDA@mail.gmail.com