Re: [report] memory leaks in COPY FROM on partitioned table

2018-08-03 12:38 GMT+09:00 Alvaro Herrera <alvherre(at)2ndquadrant(dot)com>:
> On 2018-Aug-03, Kohei KaiGai wrote:
>
>> 2018-08-02 5:38 GMT+09:00 Alvaro Herrera <alvherre(at)2ndquadrant(dot)com>:
>> > On 2018-Aug-01, Alvaro Herrera wrote:
>> >
>> >> Right, makes sense. Pushed that way.
>> >
>> > KaiGai, if you can please confirm that the pushed change fixes your test
>> > case, I'd appreciate it.
>>
>> Can you wait for a few days? I can drop the test dataset and reuse the storage
>> once benchmark test is over....
>
> Of course -- take your time.
>
I could load the same data (544GB csv, 789GB heap) using COPY FROM successfully.
When I reported the problem, rss usage of postgresql process increased
about 10MB/s ratio,
then OOM killer eliminated after a few hours.

Now, it consumed about 60MB rss at the beginning of COPY FROM, and it grows up
very slowly during the COPY FROM execution, then grew up to 250MB
before completion.
We may have another memory blocks which are not released during
execution, however,
I could not identify whether it is really memory leaking or not, and who's jobs.

It may be an idea to put a debug code that raises a notice when
MemoryContext allocates
more than the threshold.

Thanks,
--
HeteroDB, Inc / The PG-Strom Project
KaiGai Kohei <kaigai(at)heterodb(dot)com>