Re: Latin vs non-Latin words in text search parsing

Alvaro Herrera <alvherre(at)commandprompt(dot)com> writes:
> Tom Lane wrote:
>> ISTM that perhaps a more generally useful definition would be
>>
>> lword Only ASCII letters
>> nlword Entirely letters per iswalpha(), but not lword
>> word Entirely alphanumeric per iswalnum(), but not nlword

> ... how about

> lword Entirely letters per iswalpha, with at least one ASCII
> nlword Entirely letters per iswalpha
> word Entirely alphanumeric per iswalnum, but not nlword

Hmm. Then we have no category for "entirely ASCII", which is an
interesting category at least from the English standpoint, and I think
also in a lot of computer-oriented contexts. I think you may be putting
too much emphasis on the "Latin" aspect of the category name, which I
find to be a bit historical. I'm not sure if it's too late to consider
renaming the categories; if we were willing to do that I'd propose
categories "aword", "naword", "word", defined as above.

Another thing that bothers me about your suggestion is that (at least in
some locales) iswalpha will return true for things that are neither
ASCII letters nor accented versions of them, eg Cyrillic letters.
So I'm not sure the surprise factor is any less with your approach
than mine: you could still get "lword" for something decidedly not
Latin-derived.

regards, tom lane