Re: Why "copy ... from stdio" does not return immediately when reading invalid data?

From: John R Pierce <pierce(at)hogranch(dot)com>
To: pgsql-general(at)postgresql(dot)org
Subject: Re: Why "copy ... from stdio" does not return immediately when reading invalid data?
Date: 2011-02-02 19:06:01
Message-ID: 4D49AB19.4020502@hogranch.com
Views: Raw Message | Whole Thread | Download mbox | Resend email
Thread:
Lists: pgsql-general pgsql-hackers

On 02/02/11 10:20 AM, Nicolas Grilly wrote:
> Is the copy protocol (aka PQputCopyData and PQputCopyEnd) designed to
> send gigabytes of data with just one "copy ... from stdio" query, and
> is there a way to be notified of a potential error before calling
> PQputCopyEnd? Or do I have to send my data in small chunks (for
> example batch of 10000 rows), issue a PQputCopyEnd, check for errors,
> and continue with the next chunk?

I would batch the data, maybe 1000 lines or even 100 lines at a time if
these errors are at all frequent. put the errored batches in an
exception list or something so you can sort them out later.

In response to

Responses

Browse pgsql-general by date

  From Date Subject
Next Message William Bruton 2011-02-02 19:08:19 upgrade
Previous Message Carlos Mennens 2011-02-02 18:32:41 Database Design Question

Browse pgsql-hackers by date

  From Date Subject
Next Message vaibhavkaushal123 2011-02-02 19:25:12 Elephants
Previous Message Greg Smith 2011-02-02 18:47:21 Re: [HACKERS] Slow count(*) again...