Re: encoding affects ICU regex character classification

On Wed, 2023-11-29 at 18:56 -0500, Tom Lane wrote:
> We'd have to find an alternate source of knowledge to replace the
> <wctype.h> functions if we wanted to fix it fully ... can ICU do
> that?

My follow-up proposal is exactly along those lines, except that we
don't even need ICU.

By adding a couple lookup tables generated from the Unicode data files,
we can offer a pg_u_isalpha() family of functions. As a bonus, I have
some exhaustive tests to compare with what ICU does so we can protect
ourselves from simple mistakes.

I might as well send it now; patch attached (0003 is the interesting
one).

I also tested against the iswalpha() family of functions, and those
have very similar behavior (apart from the "C" locale, of course).
Character classification is not localized at all in libc or ICU as far
as I can tell.

There are some differences, and I don't understand why those
differences exist, so perhaps that's worth discussing. Some differences
seem to be related to the titlecase/uppercase distinction. Others are
strange, like how glibc counts some digit characters (outside 0-9) as
alphabetic. And some seem arbitrary, like excluding a few whitespace
characters. I can try to post more details if that would be helpful.

Another issue is that right now we are doing the wrong thing with ICU:
we should be using the u_isUAlphabetic() family of functions, not the
u_isalpha() family of functions.

Regards,
Jeff Davis