Re: Assert failure on 'list_member_ptr(rel->joininfo, restrictinfo)'

On Fri, Dec 8, 2023 at 12:43 PM Alexander Pyhalov
<a(dot)pyhalov(at)postgrespro(dot)ru> wrote:
>
> Andrei Lepikhov писал(а) 2023-12-08 07:37:
> > On 28/11/2023 01:37, Alexander Korotkov wrote:
> >> On Mon, Nov 27, 2023 at 8:07 PM Andres Freund <andres(at)anarazel(dot)de>
> >> wrote:
> > Sorry for the late answer, I missed this thread because of vacation.
> >>> On 2023-11-27 11:29:48 +0530, Ashutosh Bapat wrote:
> >>>> How do we ensure that we are not making unnecessary copies of
> >>>> Bitmapsets?
> >>>
> >>> We don't - but that's not specific to this patch. Bitmapsets
> >>> typically aren't
> >>> very large, I doubt that it's a significant proportion of the memory
> >>> usage. Adding refcounts or such would likely add more overhead than
> >>> it'd save,
> >>> both in time and memory.
> >
> > I'd already clashed with Tom on copying the required_relids field and
> > voluntarily made unnecessary copies in the project [1].
> > And ... stuck into huge memory consumption. The reason was in
> > Bitmapsets:
> > When we have 1E3-1E4 partitions and try to reparameterize a join, one
> > bitmapset field can have a size of about 1kB. Having bitmapset
> > referencing Relation with a large index value, we had a lot of (for
> > example, 1E4 * 1kB) copies on each reparametrization of such a field.
> > Alexander Pyhalov should remember that case.
>
> Yes. If it matters, this happened during reparametrization when 2
> partitioned tables with 1000 partitions each were joined. Then
> asymmetric pw join managed to eat lots of memory for bitmapsets (by
> lots of memory I mean all available on the test VM).

I did some analysis of memory consumption by bitmapsets in such cases.
[1] contains slides with the result of this analysis. The slides are
crude and quite WIP. But they will give some idea.

[1] https://docs.google.com/presentation/d/1S9BiAADhX-Fv9tDbx5R5Izq4blAofhZMhHcO1c-wzfI/edit?usp=sharing

--
Best Wishes,
Ashutosh Bapat