Re: [report] memory leaks in COPY FROM on partitioned table

On 2018-Aug-04, Kohei KaiGai wrote:

> I could load the same data (544GB csv, 789GB heap) using COPY FROM successfully.
> When I reported the problem, rss usage of postgresql process increased
> about 10MB/s ratio, then OOM killer eliminated after a few hours.

OK, I think we can consider this particular bug closed, then.

> Now, it consumed about 60MB rss at the beginning of COPY FROM, and it
> grows up very slowly during the COPY FROM execution, then grew up to
> 250MB before completion.
> We may have another memory blocks which are not released during
> execution, however,
> I could not identify whether it is really memory leaking or not, and
> who's jobs.

Most likely, this is a different memory leak.

I sugges that one way to track this down is first figure out *which*
context is the one growing, which you can see by running
MemoryContextStats a few times and noting for meaningful differences.
Then we can try to narrow down what is allocating stuff in that context.

> It may be an idea to put a debug code that raises a notice when
> MemoryContext allocates more than the threshold.

I don't think this is really practical, because whatever the threshold
we set, there would be some corner-case scenario where the threshold is
legitimately crossed. And some memory leak scenarios that don't cross
any thresholds.

--
Álvaro Herrera https://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services