Re: [PERFORMANCE] Error loading 37G CSV file "invalid string enlargement request size 65536"

From: Tom Lane <tgl(at)sss(dot)pgh(dot)pa(dot)us>
To: "Adonias Malosso" <malosso(at)gmail(dot)com>
Cc: pgsql-performance(at)postgresql(dot)org
Subject: Re: [PERFORMANCE] Error loading 37G CSV file "invalid string enlargement request size 65536"
Date: 2008-04-22 21:55:23
Message-ID: 27806.1208901323@sss.pgh.pa.us
Views: Raw Message | Whole Thread | Download mbox | Resend email
Thread:
Lists: pgsql-performance

"Adonias Malosso" <malosso(at)gmail(dot)com> writes:
> Im running a copy for a 37G CSV and receiving the following error:
> "invalid string enlargement request size 65536"

AFAICS this would only happen if you've got an individual line of COPY
data exceeding 1GB. (PG versions later than 8.2 give a slightly more
helpful "out of memory" error in such a case.)

Most likely, that was not your intention, and the real problem is
incorrect quoting/escaping in the CSV file, causing COPY to think
that a large number of physical lines should be read as one logical line.

regards, tom lane

In response to

Browse pgsql-performance by date

  From Date Subject
Next Message Ivan Voras 2008-04-22 23:50:33 Re: Suspicious top output
Previous Message Adonias Malosso 2008-04-22 21:05:35 [PERFORMANCE] Error loading 37G CSV file "invalid string enlargement request size 65536"