Re: Should CSV parsing be stricter about mid-field quotes?

On Fri, May 12, 2023, at 21:57, Andrew Dunstan wrote:
> Maybe this is unexpected by you, but it's not by me. What other sane interpretation of that data could there be? And what CSV producer outputs such horrible content? As you've noted, ours certainly does not. Our rules are clear: quotes within quotes must be escaped (default escape is by doubling the quote char). Allowing partial fields to be quoted was a deliberate decision when CSV parsing was implemented, because examples have been seen in the wild.
>
> So I don't think our behaviour is broken or needs fixing. As mentioned by Greg, this is an example of the adage about being liberal in what you accept.
>

I understand your position, and your points are indeed in line with the
traditional "Robustness Principle" (aka "Postel's Law") [1] from 1980, which
suggests "be conservative in what you send, be liberal in what you accept."
However, I'd like to offer a different perspective that might be worth
considering.

A 2021 IETF draft, "The Harmful Consequences of the Robustness Principle" [2],
argues that the flexibility advocated by Postel's Law can lead to problems such
as unclear specifications and a multitude of varying implementations. Features
that initially seem helpful can unexpectedly turn into bugs, resulting in
unanticipated consequences and data integrity risks.

Based on the feedback from you and others, I'd like to revise my earlier
proposal. Rather than adding an option to preserve the existing behavior, I now
think it's better to simply report an error in such cases. This approach offers
several benefits: it simplifies the CSV parser, reduces the risk of
misinterpreting data due to malformed input, and prevents the all-too-familiar
situation where users blindly apply an error hint without understanding the
consequences.

Finally, I acknowledge that we can't foresee the number of CSV producers that
produce mid-field quoting, and this change may cause compatibility issues for
some users. However, I consider this an acceptable tradeoff. Users encountering
the error would receive a clear message explaining that mid-field quoting is not
allowed and that they should change their CSV producer's settings to escape
quotes by doubling the quote character. Importantly, this change guarantees that
previously parsed data won't be misinterpreted, as it only enforces stricter
parsing rules.

[1] https://datatracker.ietf.org/doc/html/rfc761#section-2.10
[2] https://www.ietf.org/archive/id/draft-iab-protocol-maintenance-05.html

/Joel