Re: Unicode normalization SQL functions

On 2020-02-13 01:23, Andreas Karlsson wrote:
> A potential optimization would be to merge utf8_to_unicode() and
> pg_utf_mblen() into one function in unicode_normalize_func() since
> utf8_to_unicode() already knows length of the character. Probably not
> worth it though.

This would also require untangling the entire encoding API.

> It feels a bit wasteful to measure output_size in
> unicode_is_normalized() since unicode_normalize() actually already knows
> the length of the buffer, it just does not return it.

Sure, but really most string APIs work like that. They surely know the
string length internally, but afterwards you often have to call strlen()
again.

> A potential optimization for the normalized case would be to abort the
> quick check on the first maybe and normalize from that point on only. If
> I can find the time I might try this out and benchmark it.

Are you sure this would always be valid? The fact that this wasn't
mentioned in UTR #15 makes me suspicious.

> Nitpick: "split/\s*;\s*/, $line" in generate-unicode_normprops_table.pl
> should be "split /\s*;\s*/, $line".

done

> What about using else if in the code below for clarity?
>
> + if (check == UNICODE_NORM_QC_NO)
> + return UNICODE_NORM_QC_NO;
> + if (check == UNICODE_NORM_QC_MAYBE)
> + result = UNICODE_NORM_QC_MAYBE;

done

> Remove extra space in the line below.
>
> + else if (quickcheck == UNICODE_NORM_QC_NO )

I didn't find this in my local copy.

--
Peter Eisentraut http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services