Re: psql blows up on BOM character sequence

On 03/24/2014 02:50 PM, Jim Nasby wrote:
> On 3/22/14, 11:26 AM, Jim Nasby wrote:
>> On 3/21/14, 4:54 PM, Tom Lane wrote:
>>> Merlin Moncure <mmoncure(at)gmail(dot)com> writes:
>>>> There is no way for psql to handle that case though unless you'd strip
>>>> *all* BOMs encountered. Compounding this problem is that there's no
>>>> practical way AFAIK to send multiple file to psql via single command
>>>> line invocation. If you pass multiple -f arguments all but one is
>>>> ignored.
>>>
>>> Well, that seems like a solvable but rather independent problem.
>>> I guess one issue is how you'd define the meaning of --single ...
>>> one transaction per run, or one per file?
>>
>> Well, if you're catting multiple files into psql -1, you'd get all
>> the files in one transaction, right? So I'd say that's what should
>> happen.
>
> It occurs to me that we're going about this the wrong way...
>
> The error here isn't being generated by psql; it's generated by the
> backend. In the context of a statement (and not, say, a COPY command).
>
> So instead of trying to handle this on the psql side[1], I think we
> need to handle it in the backend; specifically in the parser. Is there
> an easy way to get the parser to ignore the BOM character in the
> context of commands (but not in strings)?
>
> [1]: Obviously, BOM could still screw up a psql command like \d. We'd
> want to address that as well; but I suspect backends are the more
> common scenario.

But what about COPY files? I don't see why there is less of a case for
eating a leading BOM for a COPY file (or COPY stdin for that matter,
given that it can come from \copy) than for an SQL file.

I suspect suspect trying to do this in the parser will be quite messy.
This needs to happen before the input is converted to the server
encoding, I think.

cheers

andrew