Re: BUG #10589: hungarian.stop file spelling error

On 11/06/14 15:09, Tom Lane wrote:
> I wrote:
>>> [ we seem to have gotten a misencoded version of hungarian.stop ]
>> Actually, it looks like things are even worse than that: the Hungarian
>> stemmer code seems to be confused about this too. In the first place,
>> we've got a LATIN1 version of that stemmer, which I would imagine is
>> entirely useless; and in the second place, the UTF8 version has no
>> reference to any non-LATIN1 characters.
>> Again, I'm suspecting this problem goes further than Hungarian,
>> because the set of stem_ISO_8859_1_foo.c files in
>> src/backend/snowball/libstemmer/ covers a lot more languages than
>> I think LATIN1 is meant to cope with. I'm not sure how much of this
>> is broken in the original Snowball code and how much is our error
>> while importing the code.
> After further analysis, it appears that:
>
> 1. The cause of the immediately complained-of problem is that we took
> the stopword file we got from the Snowball website to be in LATIN1,
> whereas it evidently was meant to be in LATIN2. The problematic
> characters were code 0xF5 in the file, which we translated to U+00F5,
> but the correct translation is U+0151. (There is another discrepancy
> between LATIN1 and LATIN2 at code point 0xFB, but by chance there are
> none of those in the stopword file.)
>
> 2. The Snowball people were just as confused as we were about the
> appropriate encoding to use for Hungarian: their code claims that the
> Hungarian stemmer can run in LATIN1, and contains this table of non-ASCII
> character codes used in it:
>
> /* special characters (in ISO Latin I) */
>
> stringdef a' hex 'E1' //a-acute
> stringdef e' hex 'E9' //e-acute
> stringdef i' hex 'ED' //i-acute
> stringdef o' hex 'F3' //o-acute
> stringdef o" hex 'F6' //o-umlaut
> stringdef oq hex 'F5' //o-double acute
> stringdef u' hex 'FA' //u-acute
> stringdef u" hex 'FC' //u-umlaut
> stringdef uq hex 'FB' //u-double acute
>
> Most of these codes are the same in LATIN1 and LATIN2, but o-double-acute
> and u-double-acute don't appear in LATIN1 at all, and the codes shown here
> are really for LATIN2.
>
> I've reported this issue upstream and there are fixes pending.
>
> 3. While I was concerned that there might be similar bugs in the other
> Snowball stemmers, it appears after a bit of research that LATIN1 is
> commonly used as an encoding for all the other languages the Snowball
> code claims it can be used for, even though in a few cases there are
> seldom-used characters that LATIN1 can't represent. So there's not a
> clear reason to think there are any other undetected problems (and
> I would certainly not be the man to find them if they exist).
>
>
> I've gone ahead and committed the encoding fix for hungarian.stop in all
> active branches. I'm going to wait for Snowball upstream to accept the
> proposed patches before I think about incorporating the code changes.
>
> I'm not real sure whether we should consider back-patching those changes.
> Right now, the Hungarian stemmer is applying rules meant for
> o-double-acute to o-tilde, which probably means that those stemming rules
> don't fire at all on actual Hungarian text. If we fix that then the
> stemmer will behave differently, which might not be all that desirable to
> change in a minor release. Perhaps we should only make the code changes
> in HEAD and 9.4?
>
> regards, tom lane
>
>
Not saying there is any problem, but you might like to check how the EUR
currency symbol is handled (it is in LATIN2, but not in LATIN1):

https://en.wikipedia.org/wiki/Euro_sign
U+20AC € euro sign
(HTML: |€| |€|)

Cheers,
Gavin