Re: How to insert bulk data with libpq in C?

From: Chris Mair <chris(at)1006(dot)org>
To: Laurenz Albe <laurenz(dot)albe(at)cybertec(dot)at>, a <372660931(at)qq(dot)com>
Cc: pgsql-general <pgsql-general(at)postgresql(dot)org>
Subject: Re: How to insert bulk data with libpq in C?
Date: 2018-12-18 10:57:09
Message-ID: 59eda4cff9a454701320e2a69569c156@smtp.hushmail.com
Views: Raw Message | Whole Thread | Download mbox | Resend email
Thread:
Lists: pgsql-general


>>>> May I ask the proper (fast) way of inserting bulk data and if possible, an example would be very appreciated.
>>>>
>>>> Please note that the data I would like to insert contains arrays (float8[]).
>>>>
>>>> By the way, my environment is Windows 10, PostgreSQL 11 and compiling with Visual Studio 2017.
>>>
>>> That would be COPY:
>>> https://www.postgresql.org/docs/current/sql-copy.html
>>
>> In particular,
>>
>> copy table (col1, col2, ...) from stdin with binary
>
> Hmm, binary requires that you know the binary PostgreSQL representation.
> PostgreSQL might be faster converting text to its internal representation
> than your application program...
>
> The big exception here is of course bytea, where the binary representation
> is definitely smaller and easy to create.

Hi,

yes.
I understood the data is already binary and OP wants to store it binary, but
of course this might not be the case.

>> Also, I suggest transaction bundling (one commit every 1000 records or so),
>> depending on context.
>
> Why that?
> It might be useful if you expect the load to fail in the middle and want
> to be able to restart from there. But normally, doing it all in one
> transaction is simpler.

Yes. That would be the special case with one commit every N records, where
N is the total count :)

I just wanted to make shure OP will not commit after each COPY.

Bye.
Chris.

In response to

Browse pgsql-general by date

  From Date Subject
Next Message Andrew Gierth 2018-12-18 13:49:55 Re: Format an Update with calculation
Previous Message Laurenz Albe 2018-12-18 10:46:10 Re: How to insert bulk data with libpq in C?