Re: BUG #17611: SJIS conversion rule about duplicated characters differ from Windows

This is not a bug, but the designed behavior. But we could change that
conversion table if a plausible reasoning is raised.

At Thu, 08 Sep 2022 11:33:17 +0000, PG Bug reporting form <noreply(at)postgresql(dot)org> wrote in
> SJIS(Windows-31J) has several defined characters that has the
> same glyph but a different code point for it. The SJIS conversion
> rules in PostgreSQL's client_encoding seem to be slightly different
> from the rules in the Windows OS.

PostgreSQL follows CP932. And no rule on the precedence between
duplicate characters is published as a public standard. According to
[2], it is published as Microsoft's recommended convention.

> In some cases, it causes a bad thing for Windows users.
> For example, some text editors can't display these characters, and
> .NET applications raise exceptions when converting SJIS byte
> sequences to UTF16 (String type). This can happen when using Npgsql[1].
>
> .NET code:
> ----
> Encoding e = Encoding.GetEncoding("shift_jis",

AFAIK generally Shift_jis and CP932 have different character sets. I
don't know about .Net but doesn't CP932 work in that case?
Specifically, "Encoding.GetEncoding(932)". There must a way to deal
with that characters since they are in CP932.

> My customers have difficulty dealing with SJIS code in Windows
> applications because of this difference in conversion rules.
> They are migrating from Oracle and many of the applications are
> written for the SJIS environment.
>
> The rules for converting from Unicode to characters that are
> duplicated in SJIS seem to be as follows in Windows[2]:
>
> 1. If the character is in both JIS X 0208 and NEC special characters,
> use the code point of JIS X 0208.
> 2. If the character is in both NEC special characters and IBM selected
> characters, use the code point of NEC special characters.
> 3. If the character is in both IBM selected characters and
> NEC selected-IBM extended characters, use the code point of
> IBM selected characters.

Mmm. I don't reach the original document by Microsoft pointed from
[2]. Could you tell me an alternative URL? (Goole didn't offer usable
info by kb170559 or somethig like)

> However, the rules for converting from Unicode to SJIS in PostgreSQL
> seem to differ from the above second rule.
> SJIS codepoints corresponding to the second rule are listed below:
> - "NEC special characters" : 0x8754 - 0x875D, 0x8782, 0x8784, 0x878A
> - "IBM selected characters": 0xFA4A - 0xFA53, 0xFA59, 0xFA5A, 0xFA58
>
> In src/backend/utils/mb/Unicode/UCS_to_SJIS.pl, @reject_sjis array
> defines the not used code points when converting Unicode to SJIS.
> According to the second rule above, the @reject_sjis array must contain
> "IBM selected characters", but it currently contains "NEC special
> characters".

Anyway it is not in the public standard and at most that "rule" is a
recommendation. So it's not the case we "must" change the conversion
table following the "rule".

FYI, the following range of SJIS character codes are *excluded* while
unicode->sjis conversion. They are not only NEC/IBM extension
characters.

ed40 - eefc : so-called "NEC extension"
uses fa40 - fc40 (IBM extension) instead.
8754 - 875d : numbers with circle, and upper roman numbers
uses fa4a - fa53 instead.
878a, 8782, 8784, fa5b, fa54: some japanese combined characters "No." "(株)"...
uses fa58, fa59, fa5a, 81e6, 879a, 81ca
8790 - 8792 : math symbols, uses 81e0, 81df, 81e7
8795 - 8797 : ditto, 81e3, 81db, 81da
879a - 879c : ditto, 879a, 81bf, 81be

> The current PostgreSQL rules for converting duplicate definition characters
>
> seems to be introduced by 5735c4cf3d059914e2b9d294203aa06fb2c4ac75,
> back in 2001, but I could not be found reason for it in past mailing list
> logs.
> I think this conversion difference is a bug,
> but is it a rule with some clear reason?

I don't know about a clear rason for the current conversion, but it is
a reason for *not* changing the conversion table that we had no
complaint about the conversion for more than ten years. Because
changing that tables could cause problems elsewhere.

> [1] https://www.npgsql.org/
> [2] https://dev.mysql.com/doc/mysql-g11n-excerpt/8.0/en/charset-cp932.html

regards.

--
Kyotaro Horiguchi
NTT Open Source Software Center