Re: Patch: add conversion from pg_wchar to multibyte

>> Tatsuo Ishii <ishii(at)postgresql(dot)org> writes:
>>>> So far as I can see, the only LCPRVn marker code that is actually in
>>>> use right now is 0x9d --- there are no instances of 9a, 9b, or 9c
>>>> that I can find.
>>>>
>>>> I also read in the xemacs internals doc, at
>>>> http://www.xemacs.org/Documentation/21.5/html/internals_26.html#SEC145
>>>> that XEmacs thinks the marker code for private single-byte charsets
>>>> is 0x9e (only) and that for private multi-byte charsets is 0x9f (only);
>>>> moreover they think 0x9a-0x9d are potential future official multibyte
>>>> charset codes. I don't know how we got to the current state of using
>>>> 0x9a-0x9d as private charset markers, but it seems pretty inconsistent
>>>> with XEmacs.
>>
>>> At the time when mule internal code was introduced to PostgreSQL,
>>> xemacs did not have multi encoding capabilty and mule (a patch to
>>> emacs) was the only implementation allowed to use multi encoding. So I
>>> used the specification of mule documented in the URL I wrote.
>>
>> I see. Given that upstream has decided that a simpler definition is
>> more appropriate, is there any reason not to follow their lead, to the
>> extent that we can do so without breaking existing on-disk data?
>
> Please let me spend week end to understand the their latest spec.

This is an intermediate report on the internal multi-byte charset
implementation of emacen. I have read the link Tom showed. Also I made
a quick scan on xemacs-21.4.0 source code, especially
xemacs-21.4.0/src/mule-charset.h. It seems the web document is
essentially a copy of the comments in the file. Also I looked into
other place of xemacs code and I think I can conclude that xeamcs
21.4's multi-byte implementation is based on the doc on the web.

Next I looked into emacs 24.1 source code because I could not find any
doc regarding emacs's(not xemacs's) implementation of internal
multi-byte charset. I found followings in src/charset.h:

/* Leading-code followed by extended leading-code. DIMENSION/COLUMN */
#define EMACS_MULE_LEADING_CODE_PRIVATE_11 0x9A /* 1/1 */
#define EMACS_MULE_LEADING_CODE_PRIVATE_12 0x9B /* 1/2 */
#define EMACS_MULE_LEADING_CODE_PRIVATE_21 0x9C /* 2/2 */
#define EMACS_MULE_LEADING_CODE_PRIVATE_22 0x9D /* 2/2 */

And these are used like this:

/* Read one non-ASCII character from INSTREAM. The character is
encoded in `emacs-mule' and the first byte is already read in
C. */

static int
read_emacs_mule_char (int c, int (*readbyte) (int, Lisp_Object), Lisp_Object readcharfun)
{
:
:
else if (len == 3)
{
if (buf[0] == EMACS_MULE_LEADING_CODE_PRIVATE_11
|| buf[0] == EMACS_MULE_LEADING_CODE_PRIVATE_12)
{
charset = CHARSET_FROM_ID (emacs_mule_charset[buf[1]]);
code = buf[2] & 0x7F;
}

As far as I can tell, this is exactly the same way how PostgreSQL
handles single private character sets: they consist of 3 bytes, and
leading byte is either 0x9a or 0x9b. Other examples regarding single
byte/multi-byte private charsets can be seen in coding.c.

As far as I can tell, it seems emacs and xemacs employes different
implementations of multi-byte charaset regarding "private"
charsets. Emacs's is same as PostgreSQL, while xemacs is not. I am
contacting to the original Mule author if he knows anything about
this.

BTW, while looking into emacs's source code, I found their charset
definitions are in lisp/international/mule-conf.el. According to the
file several new charsets has been added. Included is the patch to
follow their changes. This makes no changes to current behavior, since
the patch just changes some comments and non supported charsets.
--
Tatsuo Ishii
SRA OSS, Inc. Japan
English: http://www.sraoss.co.jp/index_en.php
Japanese: http://www.sraoss.co.jp