Re: Different length lines in COPY CSV

Tom Lane said:
> Bruce Momjian <pgman(at)candle(dot)pha(dot)pa(dot)us> writes:
>> Christopher Kings-Lynne wrote:
>>> Is there any way to force COPY to accept that there will be lines of
>>> different length in a data file?
>
>> I suppose we could have a TRAILINGNULL flag to COPY but because few
>> ask for this feature, it hardly seems worth it.
>
> There is no chance that we'll ever be able to cope with every insane
> file format that some benighted program claims is CSV. The harder we
> try, the more we will lose the ability to detect data errors at all;
> not to mention the likely negative consequences for the readability and
> performance of the COPY code. I think "fix it with a perl script" is a
> very reasonable answer for cases like this one.
>

I agree.

The COPY code is probably on the edge of maintainability now.

Our CSV routines accept a wide variety of imports formats, but a fixed
number of columns is required. Maybe we need a pgfoundry project with some
general perl CSV munging utilities - this issue comes up often enough.

cheers

andrew