Re: Should CSV parsing be stricter about mid-field quotes?

On Sun, Oct 6, 2024, at 15:12, Andrew Dunstan wrote:
> On 2024-10-04 Fr 12:19 PM, Joel Jacobson wrote:
>> 2. Avoid needing hacks like using E'\x01' as quoting char.
>>
>> Introduce QUOTE NONE and DELIMITER NONE,
>> to allow raw lines to be imported "as is" into a single text column.
>
> As I think I previously indicated, I'm perfectly happy about 2, because
> it replaces a far from obvious hack, but I am at best dubious about 1.

I've looked at how to implement this, and there is quite a lot of complexity
having to do with quoting and escaping.

Need guidance on what you think would be best to do:

2a) Should we aim to support all NONE combinations, at the cost of increasing the
complexity at all code having to do with quoting, escaping and delimiters?

2b) Should we aim to only support the QUOTE NONE DELIMITER NONE ESCAPE NONE case,
useful to the real-life scenario we've identified, that is, importing raw log
lines into a single column, which could then be handed by a much simpler and
probably faster version of CopyReadAttributesCSV(),
e.g. named CopyReadAttributesUnquotedUnDelimited() or
maybe CopyReadAttributesRaw()?
(We also need to modify CopyReadLineText(), but seems we only need a
quote_none bool, to skip over the quoting code there, so don't think a
separate function is warranted there.)

I think ESCAPE NONE should be implied from QUOTE NONE, since the default escape
character is the same as the quote character, so if there isn't any
quote character, then I think that would imply no escape character either.

Can we think of any other valid, useful, realistic, and safe combinations of
QUOTE NONE, DELIMITER NONE and ESCAPE NONE, that would be interesting
to support?

If not, then I think 2b looks more interesting, to reduce risk of accidental
misuse, simpler implementation, and since it also should allow importing
raw log files faster, thanks to the reduced complexity.

Best regards,

Joel