From: | Adrian Klaver <adrian(dot)klaver(at)aklaver(dot)com> |
---|---|
To: | Jim Nasby <Jim(dot)Nasby(at)BlueTreble(dot)com>, "psycopg(at)postgresql(dot)org" <psycopg(at)postgresql(dot)org> |
Subject: | Re: Turbo ODBC |
Date: | 2017-01-15 21:25:11 |
Message-ID: | e98422f7-02a7-691e-c11b-d12e217b445c@aklaver.com |
Views: | Raw Message | Whole Thread | Download mbox | Resend email |
Thread: | |
Lists: | psycopg |
On 01/15/2017 01:19 PM, Jim Nasby wrote:
> On 1/15/17 3:04 PM, Adrian Klaver wrote:
>> On 01/15/2017 12:32 PM, Jim Nasby wrote:
>>> Interesting... https://github.com/blue-yonder/turbodbc
>>
>> Yes, interesting but you are still dependent on the underlying ODBC
>> implementation. Not sure how that impacts performance versus a method
>> with fewer hops?
>
> Oh, I'd hope that a native libpq implementation would be faster than
> going through ODBC. But, there's presumably useful info that can be
> picked up here; the bit about buffering was certainly interesting.
>
> BTW, the person that brought this to my attention had mentioned that a
> lot of people doing data science with data living in Postgres feel the
> need to extract the data from Postgres into something like HDFS before
> they can do anything useful, because apparently data access through HDFS
> is 3x faster than through Postgres. My impression is that at least part
Have you looked at asyncpg:
https://github.com/MagicStack/asyncpg
it is Python 3.5+ though.
> of that is due to using Pandas from_sql functionality (which AIUI
> marshals everything through SQL Alchemy), but anything that can be done
> on the psycopg2 side would help.
>
> I'm also looking into speeding up SPI access through plpython; depending
> on how you want to measure I've gotten a 30-600% improvement by removing
> the buffering that SPI does by default.
--
Adrian Klaver
adrian(dot)klaver(at)aklaver(dot)com
From | Date | Subject | |
---|---|---|---|
Next Message | Jim Nasby | 2017-01-16 23:14:51 | Re: Turbo ODBC |
Previous Message | Jim Nasby | 2017-01-15 21:19:33 | Re: Turbo ODBC |