Re: Pre-proposal: unicode normalized text

On Wed, 2023-10-11 at 08:56 +0200, Peter Eisentraut wrote:
> We need to be careful about precise terminology.  "Valid" has a
> defined
> meaning for Unicode.  A byte sequence can be valid or not as UTF-8. 
> But
> a string containing unassigned code points is not not-"valid" as
> Unicode.

New patch attached, function name is "unicode_assigned".

I believe the patch has utility as-is, but I've been brainstorming a
few more ideas that could build on it:

* Add a per-database option to enforce only storing assigned unicode
code points.

* (More radical) Add a per-database option to normalize all text in
NFC.

* Do character classification in Unicode rather than relying on
glibc/ICU. This would affect regex character classes, etc., but not
affect upper/lower/initcap nor collation. I did some experiments and
the General Category doesn't change a lot: a total of 197 characters
changed their General Category since Unicode 6.0.0, and only 5 since
ICU 11.0.0. I'm not quite sure how to expose this, but it seems like a
nicer way to handle it than tying it into the collation provider.

Regards,
Jeff Davis