Re: Optimize planner memory consumption for huge arrays

From: "Lepikhov Andrei" <a(dot)lepikhov(at)postgrespro(dot)ru>
To: "Dilip Kumar" <dilipbalaut(at)gmail(dot)com>
Cc: pgsql-hackers(at)lists(dot)postgresql(dot)org, Евгений Бредня <e(dot)brednya(at)postgrespro(dot)ru>
Subject: Re: Optimize planner memory consumption for huge arrays
Date: 2023-09-04 10:19:05
Message-ID: 8498b480-51f8-4827-ac3a-c0c87872810c@app.fastmail.com
Views: Raw Message | Whole Thread | Download mbox | Resend email
Thread:
Lists: pgsql-hackers

On Mon, Sep 4, 2023, at 3:37 PM, Dilip Kumar wrote:
> On Mon, Sep 4, 2023 at 11:58 AM Lepikhov Andrei
> <a(dot)lepikhov(at)postgrespro(dot)ru> wrote:
>>
>> Hi, hackers,
>>
>> Looking at the planner behaviour with the memory consumption patch [1], I figured out that arrays increase memory consumption by the optimizer significantly. See init.sql in attachment.
>> The point here is that the planner does small memory allocations for each element during estimation. As a result, it looks like the planner consumes about 250 bytes for each integer element.
>>
>> It is maybe not a problem most of the time. However, in the case of partitions, memory consumption multiplies by each partition. Such a corner case looks weird, but the fix is simple. So, why not?
>>
>> The diff in the attachment is proof of concept showing how to reduce wasting of memory. Having benchmarked a bit, I didn't find any overhead.
>
> + Const *c = makeConst(nominal_element_type,
> + -1,
> + nominal_element_collation,
> + elmlen,
> + elem_values[i],
> + elem_nulls[i],
> + elmbyval);
> +
> + args = list_make2(leftop, c);
> if (is_join_clause)
> s2 = DatumGetFloat8(FunctionCall5Coll(&oprselproc,
> clause->inputcollid,
> @@ -1984,7 +1985,8 @@ scalararraysel(PlannerInfo *root,
> ObjectIdGetDatum(operator),
> PointerGetDatum(args),
> Int32GetDatum(varRelid)));
> -
> + list_free(args);
> + pfree(c);
>
> Maybe you can just use list_free_deep, instead of storing the constant
> in a separate variable.
As I see, the first element in the array is leftop, which is used on other iterations. So, we can't use list_free_deep here.

--
Regards,
Andrei Lepikhov

In response to

Responses

Browse pgsql-hackers by date

  From Date Subject
Next Message Jeevan Chalke 2023-09-04 10:21:22 Re: More new SQL/JSON item methods
Previous Message vignesh C 2023-09-04 10:15:07 Re: persist logical slots to disk during shutdown checkpoint