Re: [report] memory leaks in COPY FROM on partitioned table

2018-08-06 1:50 GMT+09:00 Alvaro Herrera <alvherre(at)2ndquadrant(dot)com>:
>> Now, it consumed about 60MB rss at the beginning of COPY FROM, and it
>> grows up very slowly during the COPY FROM execution, then grew up to
>> 250MB before completion.
>> We may have another memory blocks which are not released during
>> execution, however,
>> I could not identify whether it is really memory leaking or not, and
>> who's jobs.
>
> Most likely, this is a different memory leak.
>
> I sugges that one way to track this down is first figure out *which*
> context is the one growing, which you can see by running
> MemoryContextStats a few times and noting for meaningful differences.
> Then we can try to narrow down what is allocating stuff in that context.
>
Yes, but the hardest is identification of which memory context is growing
up time by time. Once we could identify, MemoryContextStats() tells us
the name of problematic context and details.

Of course, above my observation is just based on rss usage of postgresql.
It can increase physical page allocation by page fault on the virtual address
space correctly allocated.

>> It may be an idea to put a debug code that raises a notice when
>> MemoryContext allocates more than the threshold.
>
> I don't think this is really practical, because whatever the threshold
> we set, there would be some corner-case scenario where the threshold is
> legitimately crossed. And some memory leak scenarios that don't cross
> any thresholds.
>
I assume this threshold is configurable by GUC, and disabled on the default.
Once a user found suspicious memory usage increase, we can set a threshold
value. In above case, we may be able to see something around 120MB threshold.

Thanks,
--
HeteroDB, Inc / The PG-Strom Project
KaiGai Kohei <kaigai(at)heterodb(dot)com>