Re: COPY from STDIN vs file with large CSVs

From: Ron <ronljohnsonjr(at)gmail(dot)com>
To: pgsql-admin(at)lists(dot)postgresql(dot)org
Subject: Re: COPY from STDIN vs file with large CSVs
Date: 2020-01-08 17:08:56
Message-ID: c2e587ae-d58d-8d21-0a21-78c323867d08@gmail.com
Views: Raw Message | Whole Thread | Download mbox | Resend email
Thread:
Lists: pgsql-admin

On 1/8/20 10:54 AM, Wells Oliver wrote:
> I have a CSV that's ~30GB. Some 400m rows. Would there be a meaningful
> performance difference to run COPY from STDIN using: cat f.csv | psql
> "COPY .. FROM STDIN WITH CSV" versus just doing "COPY ... FROM 'f.csv'
> WITH CSV"?
>
> Thanks. It took about four hours to copy one and I felt that was a little
> much.

catting the file starts another process, and opens a pipe.  That can't be
faster than "COPY ... FROM ... WITH CSV".

pg_bulkload (which might be in your repository) is probably what you really
want.

--
Angular momentum makes the world go 'round.

In response to

Responses

Browse pgsql-admin by date

  From Date Subject
Next Message Scott Ribe 2020-01-08 17:49:07 Re: COPY from STDIN vs file with large CSVs
Previous Message Indirani Venkopa Chetty 2020-01-08 17:05:17 RE: COPY from STDIN vs file with large CSVs