Re: Testing of a fast method to bulk insert a Pandas DataFrame into Postgres

Hello, Adrian Klaver,
What is the robust way to upgrade Pandas?
Regards,
David

On Monday, 4 October 2021, Adrian Klaver <adrian(dot)klaver(at)aklaver(dot)com> wrote:

> On 10/4/21 9:20 AM, Shaozhong SHI wrote:
>
>> Hello, Adrian Klaver,
>>
>> Pandas version is 0.23.0.
>>
>
> The reason the below does not work is method did not show up until pandas
> 0.24.0.
>
>
>> I used the following code:
>>
>> def psql_insert_copy(table, conn, keys, data_iter):
>> # gets a DBAPI connection that can provide a cursor
>> dbapi_conn = conn.connection
>> with dbapi_conn.cursor() as cur:
>> s_buf = StringIO()
>> writer = csv.writer(s_buf)
>> writer.writerows(data_iter)
>> s_buf.seek(0)
>>
>> columns = ', '.join('"{}"'.format(k) for k in keys)
>> if table.schema:
>> table_name = '{}.{}'.format(table.schema, table.name <
>> http://table.name>)
>> else:
>> table_name = table.name <http://table.name>
>>
>> sql = 'COPY {} ({}) FROM STDIN WITH CSV'.format(
>> table_name, columns)
>> cur.copy_expert(sql=sql, file=s_buf)
>> engine = create_engine('postgresql+psycopg2://:5432/postgres')
>> try:
>> df.to_sql('test1', engine, schema='public', if_exists='append',
>> index=False, method=psql_insert_copy)
>>
>> I could not find obvious reasons.
>>
>
>
>
>> Regards,
>>
>> David
>>
>>
>
> --
> Adrian Klaver
> adrian(dot)klaver(at)aklaver(dot)com
>