Re: UTF8 with BOM support in psql

Peter Eisentraut wrote:
> On ons, 2009-10-21 at 13:11 +0900, Itagaki Takahiro wrote:
>
>> Client encoding is declared in body of a file, but BOM is
>> in head of the file. So, we should always ignore BOM sequence
>> at the file head no matter what client encoding is used.
>>
>> The attached patch replace BOM with while spaces, but it does not
>> change client encoding automatically. I think we can always ignore
>> client encoding at the replacement because SQL command cannot start
>> with BOM sequence. If we don't ignore the sequence, execution of
>> the script must fail with syntax error.
>>
>
> I don't know what the best solution is here. The BOM encoded as UTF-8
> is valid data in other encodings. Of course, there is your point that
> such data cannot be at the start of an SQL command.
>
> There is also the notion of how files are handled on Unix. Normally,
> you'd assume that all of
>
> psql -f file.sql
> psql < file.sql
> cat file.sql | psql
> cat file1.sql file2.sql | psql
>
> behave consistently. That would require that the BOM is ignored in the
> middle of the data stream (which is legal and required per Unicode
> standard) and that this only happens if the character set is actually
> Unicode.
>
>
>

Cases 2 and 3 should be indistinguishable from psql's POV, although case
3 wins a "Useless Use of cat" award.

If we are only eating a BOM at the start of a file, which was the
consensus IIRC, and we treat STDIN as a file for this purpose, then we
would eat the leading BOM on file.sql and file1.sql in all the cases
above but not on file2.sql since we would not have any idea where the
file boundary was. That last case strikes me as a not very likely usage
(I'm pretty sure I've never used it, at least). A file containing:

\i file1.sql
\i file2.sql

would be the workaround if needed.

As for handling the fact that client encoding can't be set in a script
until after the leading BOM, there is always

PGOPTIONS="-c client_encoding=UTF8"

or similar.

cheers

andrew