Re: Windows UTF8 system locale

On Tue, Dec 17, 2024 at 02:29:59AM +1300, Thomas Munro wrote:
> On Sun, Dec 15, 2024 at 3:32 PM Noah Misch <noah(at)leadboat(dot)com> wrote:
> > For PostgreSQL, I expect the most obvious problems will arise for rolname and
> > datname containing non-UTF8. For example, pg_dumpall relies on
> > appendShellString() to call pg_dump for arbitrary datname. pg_dumpall would
> > get "database ... does not exist".
>
> Right, those catalogues have undefined encoding (the initial problem
> my CLUSTER ENCODING proposal started trying to fix) and could even be
> different for every row, and Windows wants all strings used in
> non-wide environ, argv, file APIs, etc to be valid in the ACP (because
> it converts them to UTF-16). We would get away with it if UTF-8
> weren't so picky, but come to think of it, so is SJIS, so maybe this
> is not a new problem with $SUBJECT?
>
> Wild guess: 文字化け (= mojibake) when encoded as UTF-8 and then passed in
> a command line to CreateProcess() with ACP=SJIS might show the problem
> (I just gave that string to iconv -f SJIS -t UTF-8 and it rejected it,
> I'm assuming that means it'd do the same sort of thing in that
> context).

I wasn't ready to believe it, but 010_dump_connstr indeed fails with
GetACP()==932. We've had test coverage of this for 8+ years, so I gather few
or no runs of the TAP suite on GetACP()==932 systems have ever happened. Wow.

Here's how your particular example traverses the CP932 command line:

CreateProcessA(0xe6 0x96 0x87 0xe5 0xad 0x97 0xe5 0x8c 0x96 0xe3 0x81 0x91)
argv[1] = e6 96 81 45 ad 97 e5 8c 96 e3 81
GetCommandLineA() = 61 20 e6 96 81 45 ad 97 e5 8c 96 e3 81
GetCommandLineW() = 61 20 8b41 30fb ff6d 601c 55a7 7e3a

> It's a shame the implicit conversion here doesn't fail with EILSEQ. I
> can't imagine how anything good can ever have come from lossy,
> non-error-raising implicit conversions anywhere near argv[]. On the

It's a shame.

> other hand, on Unix we have other problems stemming from the
> undefinedness. What does "copy ... to '/tmp/café.txt" do inside a
> LATIN1 database? macOS: EILSEQ, can't open that file, Linux: sure,
> now you have a file whose name is displayed as caf�.txt in your UTF-8
> terminal or other software (U+FFFD REPLACEMENT CHARACTER).

GNU ls provides nine options for rendering that name to a terminal:
https://www.gnu.org/software/coreutils/manual/html_node/Formatting-the-file-names.html
https://www.gnu.org/software/coreutils/quotes.html

Non-default option "ls --quoting=literal" does display the "replacement
character" way. It may count as a shame that POSIX pathnames are [0x1,0xFF]
binary strings instead of Unicode character strings, but here we are.

> > 2. Just fail if the system option is enabled and we would appendShellString()
> > a non-UTF8 value.
>
> I guess the general version is just: fail if the string is not valid
> in the ACP (MB_ERR_INVALID_CHARS).

Roughly that.

> With the ACP-matching idea for CLUSTER ENCODING, it *think* it should
> become unreachable in the two recommended modes: either those strings
> would be pure ASCII, or they'd be in database encoding (same encoding
> for all databases enforced) and the ACP would match, so it would all
> be aligned without any new conversions being required. It also has an
> UNDEFINED mode so a failed encoding validation there would still be
> reachable that way. Still thinking about it all though.

I see. Interesting. Considering you need to be root to change the ACP, I'm
disinclined to bet big on requiring the ACP to match anything about encodings
used in PostgreSQL. We might get away with it, but it sounds bad for the
Poker Tracker use case.