Re: Using per-transaction memory contexts for storing decoded tuples

On Fri, Sep 20, 2024 at 10:53 PM Masahiko Sawada <sawada(dot)mshk(at)gmail(dot)com> wrote:
>
> On Thu, Sep 19, 2024 at 10:46 PM Amit Kapila <amit(dot)kapila16(at)gmail(dot)com> wrote:
> >
> > On Fri, Sep 20, 2024 at 5:13 AM David Rowley <dgrowleyml(at)gmail(dot)com> wrote:
> > >
> > > On Fri, 20 Sept 2024 at 05:03, Masahiko Sawada <sawada(dot)mshk(at)gmail(dot)com> wrote:
> > > > I've done other benchmarking tests while changing the memory block
> > > > sizes from 8kB to 8MB. I measured the execution time of logical
> > > > decoding of one transaction that inserted 10M rows. I set
> > > > logical_decoding_work_mem large enough to avoid spilling behavior. In
> > > > this scenario, we allocate many memory chunks while decoding the
> > > > transaction and resulting in calling more malloc() in smaller memory
> > > > block sizes. Here are results (an average of 3 executions):
> > >
> > > I was interested in seeing the memory consumption with the test that
> > > was causing an OOM due to the GenerationBlock fragmentation.
> > >
> >
> > +1. That test will be helpful.
>
> Sure. Here are results of peak memory usage in bytes reported by
> MemoryContextMemAllocated() (when rb->size shows 43MB):
>
> 8kB: 52,371,328
> 16kB: 52,887,424
> 32kB: 53,428,096
> 64kB: 55,099,264
> 128kB: 86,163,328
> 256kB: 149,340,032
> 512kB: 273,334,144
> 1MB: 523,419,520
> 2MB: 1,021,493,120
> 4MB: 1,984,085,888
> 8MB: 2,130,886,528
>
> Probably we can increase the size to 64kB?
>

Yeah, but before deciding on a particular size, we need more testing
on different platforms as suggested by David.

--
With Regards,
Amit Kapila.