Re: Understanding memory usage

On Wed, Oct 30, 2013 at 7:27 PM, Daniele Varrazzo <
daniele(dot)varrazzo(at)gmail(dot)com> wrote:

>
> What to you mean as "freed"? Have you deleted the cursor and made sure the
> gc reclaimed it? The cursor doesn't destroy the internal data until it is
> deleted or another query is run (because after fetchall() you can invoke
> scroll(0) and return it to Python again). And of course when the data
> returned by fetch() is released depends on the client usage.
>

By "freed", I mean doing like in the bug report #78:

del data
areadcur.close()
acon.close()
del areadcur
del acon

After a big query you may see memory usage going down as soon as you
> execute "select 1 from false" because the result is replaced by a smaller
> one.
>

That's *not* the result that I get. Doing a query returning 2 millions rows
followed by a "SELECT 1" has no effect on RSS memory usage in my case.

The only "problem" you may attribute to Psycopg is if you find an unbound
> usage of the memory. If you run some piece of code in a loop and see memory
> increasing linearly you have found a leak. Otherwise you can attribute the
> artefacts you see to the Python GC.
>

Indeed, there's no memory leak that I can see. But don't you find strange
that Python / Psycopg memory management differs between 2 roughly
equivalent query:

- a query returning 20 rows × 10 MB each
- a query returning 2 millions rows × 100 bytes each

As far as I could test, in my environment, they're clearly not equal in
terms of side effects.
For the first, I can reclaim the memory after getting the results. For the
second, I can't.

--
Damiano Albani