Understanding memory usage

From: Damiano Albani <damiano(dot)albani(at)gmail(dot)com>
To: psycopg(at)postgresql(dot)org
Subject: Understanding memory usage
Date: 2013-10-28 21:28:56
Message-ID: CAKys952SOUe+Xu+0UOUTcGBfQHZP_rKGfNKc_4EBk1hkTDQpJg@mail.gmail.com
Views: Raw Message | Whole Thread | Download mbox | Resend email
Thread:
Lists: psycopg

Hello,

I'm trying to understand how Psycopg behaves memory-wise.
Basically, in my use case, it looks like memory is not released as I
thought it would be.

I've taken the script mentioned in bug report
#78<http://psycopg.lighthouseapp.com/projects/62710/tickets/78-memory-issue>and
run on my system (Ubuntu 12.04/x64, PostgreSQL 9.3, Python 2.7.3,
Psycopg 2.5.1).

Using the original query:
SELECT pagecontent FROM test WHERE siteid = 3;

loaded 0
VmSize: 3251264 kB
cleared 0
VmSize: 112924 kB
loaded 1
VmSize: 3250496 kB
cleared 1
VmSize: 112928 kB

So far, so good -- incidentally, the lowest memory usage seems to 112 MB,
where it is reported at 78 MB in bug report #78. Any idea why?

Now, using this very simple query (column1 being a NUMBER, around 2
millions rows):
SELECT column1 FROM table1

loaded 0
VmSize: 259392 kB
cleared 0
VmSize: 232960 kB
loaded 1
VmSize: 259392 kB
cleared 1
VmSize: 226676 kB

So why isn't the memory reclaimed as in the first case?

I've also done some tests where the fetchall() call is commented
altogether. The results are :
SELECT pagecontent FROM test WHERE siteid = 3;

loaded 0
VmSize: 2226860 kB
cleared 0
VmSize: 112920 kB
loaded 1
VmSize: 2226488 kB
cleared 1
VmSize: 112924 kB

SELECT column1 FROM table1

loaded 0
VmSize: 159772 kB
cleared 0
VmSize: 113036 kB
loaded 1
VmSize: 162356 kB
cleared 1
VmSize: 112920 kB

In that case, the memory is effectively reclaimed for the second query --
getting to the lowest 112 MB number I have mentioned before.

But what is the reason why Psycopg needs memory to just run a query, *
without* fetching results?

Thanks for your help!

--
Damiano Albani

Responses

Browse psycopg by date

  From Date Subject
Next Message Daniele Varrazzo 2013-10-28 23:23:11 Re: Understanding memory usage
Previous Message Peter Eisentraut 2013-10-17 19:40:58 Re: psycopg2 doesn't use LDFLAGS from pg_config