Memory leak in plpython3u (with testcase and patch)

From: Mat Arye <mat(at)timescale(dot)com>
To: pgsql-hackers <pgsql-hackers(at)postgresql(dot)org>
Subject: Memory leak in plpython3u (with testcase and patch)
Date: 2025-01-10 18:43:00
Message-ID: CADsUR0DvVgnZYWwnmKRK65MZg7YLUSTDLV61qdnrwtrAJgU6xw@mail.gmail.com
Views: Raw Message | Whole Thread | Download mbox | Resend email
Thread:
Lists: pgsql-hackers

Hi all,

I found a memory leak in plpython3u. It happens when converting the
argument for an spi plan execution (via plpy.execute() or similar). I've
attached a sql file to create two plpython3u functions to reproduce the
issue (memory_leak_test.sql).

When running
```
set client_min_messages = 'debug';
select test_mem(1000);
```
You'll see that the memory goes to 720MB.

In contrast the test_mem_query function you won't see the leak (it uses
~70MB):
```
set client_min_messages = 'debug';
select test_mem_query(1000);
```
This is because test_mem_query executes the query using a sql string
instead of a spi plan with parameters.

I believe the issue is in the call to `PLy_output_convert` inside
`PLy_spi_execute_plan`. I've attached a patch that reduces the memory usage
to ~70MB from ~720 MB when using test_mem. I based what I did on what
`PLy_input_convert` does. But I am not an expert in this part of the code
so I am not sure the patch is correct. In particular, I don't fully grasp
the comment in `PLy_input_convert` about why the scratch is reset before
not after the conversion cycle and what that has to do with python
refcounts. A review by an expert would be appreciated.

Thanks,
Mat

--
Mat Arye, Technical lead of all thing AI @ Timescale

Attachment Content-Type Size
memory_leak_test.sql application/octet-stream 2.5 KB
memory_leak.patch application/octet-stream 1.0 KB

Responses

Browse pgsql-hackers by date

  From Date Subject
Next Message Andres Freund 2025-01-10 18:43:14 Re: Reorder shutdown sequence, to flush pgstats later
Previous Message Andres Freund 2025-01-10 18:42:11 Re: Reorder shutdown sequence, to flush pgstats later