Re: Using per-transaction memory contexts for storing decoded tuples

On Thu, Sep 19, 2024 at 10:46 PM Amit Kapila <amit(dot)kapila16(at)gmail(dot)com> wrote:
>
> On Fri, Sep 20, 2024 at 5:13 AM David Rowley <dgrowleyml(at)gmail(dot)com> wrote:
> >
> > On Fri, 20 Sept 2024 at 05:03, Masahiko Sawada <sawada(dot)mshk(at)gmail(dot)com> wrote:
> > > I've done other benchmarking tests while changing the memory block
> > > sizes from 8kB to 8MB. I measured the execution time of logical
> > > decoding of one transaction that inserted 10M rows. I set
> > > logical_decoding_work_mem large enough to avoid spilling behavior. In
> > > this scenario, we allocate many memory chunks while decoding the
> > > transaction and resulting in calling more malloc() in smaller memory
> > > block sizes. Here are results (an average of 3 executions):
> >
> > I was interested in seeing the memory consumption with the test that
> > was causing an OOM due to the GenerationBlock fragmentation.
> >
>
> +1. That test will be helpful.

Sure. Here are results of peak memory usage in bytes reported by
MemoryContextMemAllocated() (when rb->size shows 43MB):

8kB: 52,371,328
16kB: 52,887,424
32kB: 53,428,096
64kB: 55,099,264
128kB: 86,163,328
256kB: 149,340,032
512kB: 273,334,144
1MB: 523,419,520
2MB: 1,021,493,120
4MB: 1,984,085,888
8MB: 2,130,886,528

Probably we can increase the size to 64kB?

Regards,

--
Masahiko Sawada
Amazon Web Services: https://aws.amazon.com