Re: speed up unicode decomposition and recomposition

John Naylor wrote:

> I'd be curious how it compares to ICU now

I've made another run of the test in [1] with your v2 patches
from this thread against icu_ext built with ICU-67.1.
The results show the times in milliseconds to process
about 10 million short strings:

operation | unpatched | patched | icu_ext
------------+-----------+---------+---------
nfc check | 7968 | 5989 | 4076
nfc conv | 700894 | 15163 | 6808
nfd check | 16399 | 9852 | 3849
nfd conv | 17391 | 10916 | 6758
nfkc check | 8259 | 6092 | 4301
nfkc conv | 700241 | 15354 | 7034
nfkd check | 16585 | 10049 | 4038
nfkd conv | 17587 | 11109 | 7086

The ICU implementation still wins by a large margin, but
the improvements brought by the patch are gorgeous,
especially for the conversion to NFC/NFKC.
This test ran on a slower machine than what I used for [1], so
that's why all queries take longer.

For the two queries upthread, I get this:

1)
select count(normalize(t, NFC)) from (
select md5(i::text) as t from
generate_series(1,100000) as i
) s;
count
--------
100000
(1 row)

Time: 371.043 ms

VS ICU:

select count(icu_normalize(t, 'NFC')) from (
select md5(i::text) as t from
generate_series(1,100000) as i
) s;
count
--------
100000
(1 row)

Time: 125.809 ms

2)
select count(normalize(t, NFC)) from (
select repeat(U&'\00E4\00C5\0958\00F4\1EBF\3300\1FE2\3316\2465\322D', i % 3
+ 1) as t from
generate_series(1,100000) as i
) s;
count
--------
100000
(1 row)
Time: 428.214 ms

VS ICU:

select count(icu_normalize(t, 'NFC')) from (
select repeat(U&'\00E4\00C5\0958\00F4\1EBF\3300\1FE2\3316\2465\322D', i % 3
+ 1) as t from
generate_series(1,100000) as i
) s;
count
--------
100000
(1 row)

Time: 147.713 ms

[1]
https://www.postgresql.org/message-id/2c5e8df9-43b8-41fa-88e6-286e8634f00a%40manitou-mail.org

Best regards,
--
Daniel Vérité
PostgreSQL-powered mailer: https://www.manitou-mail.org
Twitter: @DanielVerite