Re: [POC] verifying UTF-8 using SIMD instructions

On Tue, Feb 9, 2021 at 4:22 PM Heikki Linnakangas <hlinnaka(at)iki(dot)fi> wrote:
>
> On 09/02/2021 22:08, John Naylor wrote:
> > Maybe there's a smarter way to check for zeros in C. Or maybe be more
> > careful about cache -- running memchr() on the whole input first might
> > not be the best thing to do.
>
> The usual trick is the haszero() macro here:
> https://graphics.stanford.edu/~seander/bithacks.html#ZeroInWord. That's
> how memchr() is typically implemented, too.

Thanks for that. Checking with that macro each loop iteration gives a small
boost:

v1, but using memcpy()

mixed | ascii
-------+-------
601 | 129

with haszero()

mixed | ascii
-------+-------
583 | 105

remove zero-byte check:

mixed | ascii
-------+-------
588 | 93

--
John Naylor
EDB: http://www.enterprisedb.com