| From: | Damiano Albani <damiano(dot)albani(at)gmail(dot)com> |
|---|---|
| To: | psycopg <psycopg(at)postgresql(dot)org> |
| Subject: | Re: Understanding memory usage |
| Date: | 2013-11-01 18:11:45 |
| Message-ID: | CAKys9514yvi9EcSccpLGtGzETfNE--jCwqBiJBy_6iFJ9-pwsA@mail.gmail.com |
| Views: | Raw Message | Whole Thread | Download mbox | Resend email |
| Thread: | |
| Lists: | psycopg |
On Thu, Oct 31, 2013 at 12:01 PM, Daniele Varrazzo <
daniele(dot)varrazzo(at)gmail(dot)com> wrote:
>
> I easily expect a much bigger overhead in building millions of Python
> object compared to building 20. Not only for the 37 bytes of overhead
> each string has (sys.getsizeof()), but also for the consequences for
> the GC to manage objects in the millions.
>
For the record, I've eventually settled for a solution using
*pgnumpy*<http://code.google.com/p/pgnumpy/>
.
It's capable of handling results made of millions of rows with very little
overhead as far as I could see.
As my original goal was to feed the data to Pandas down the line, pgnumpy
seems spot on.
--
Damiano Albani
| From | Date | Subject | |
|---|---|---|---|
| Next Message | Denis Papathanasiou | 2013-11-05 20:14:31 | Best strategy for bulk inserts where some violate unique constraint? |
| Previous Message | Daniele Varrazzo | 2013-10-31 11:01:23 | Re: Understanding memory usage |