Re: Reducing the overhead of NUMERIC data

On 11/4/05, Tom Lane <tgl(at)sss(dot)pgh(dot)pa(dot)us> wrote:
> Martijn van Oosterhout <kleptog(at)svana(dot)org> writes:
> > Yeah, and while one way of removing that dependance is to use ICU, that
> > library wants everything in UTF-16.
>
> Really? Can't it do UCS4 (UTF-32)? There's a nontrivial population
> of our users that isn't satisfied with UTF-16 anyway, so if that really
> is a restriction of ICU then we're going to have to look elsewhere :-(

The correct question to ask is something like "Does it support non-bmp
characters?" or "Does it really support UTF-16 or just UCS2?"

UTF-16 is (now) a variable width encoding which is a strict superset
of UCS2 which allows the representation of all Unicode characters.
UCS2 is fixed width and only supports characters from the basic
multilingual plane. UTF-32 and UCS4 are (now) effectively the same
thing and can represent all unicode characters with a 4 byte fixed
length word.

The code can demand UTF-16 but still be fine for non-BMP characters.
However, many things which claim to support UTF-16 really only support
UCS2 or at least have bugs in their handling of non-bmp characters.
Software that supports UTF-8 is somewhat more likely to support
non-bmp characters correctly since the variable length code paths get
more of a workout in many environments. :)