Re: Bug with Tsearch and tsvector

"Donald Fraser" <postgres(at)kiwi-fraser(dot)net> writes:
> Using the default tsearch configuration, for 'english', text is being wrongly parsed into the tsvector type.

ts_debug shows that it's being parsed like this:

alias | description | token | dictionaries | dictionary | lexemes
-----------------+---------------------------------+----------------------------------------+----------------+--------------+------------------------------------------
tag | XML tag | <span lang="EN-GB"> | {} | |
protocol | Protocol head | http:// | {} | |
url | URL | www.harewoodsolutions.co.uk/press.aspx | {simple} | simple | {www.harewoodsolutions.co.uk/press.aspx}
host | Host | www.harewoodsolutions.co.uk | {simple} | simple | {www.harewoodsolutions.co.uk}
url_path | URL path | /press.aspx</span><span | {simple} | simple | {/press.aspx</span><span}
blank | Space symbols | | {} | |
asciiword | Word, all ASCII | lang | {english_stem} | english_stem | {lang}
... etc ...

ie the critical point seems to be that url_path is willing to soak up a
string containing "<" and ">", so the span tags don't get recognized as
separate lexemes. While that's "obviously" the wrong thing in this
particular example, I'm not sure if it's the wrong thing in general.
Can anyone comment on the frequency of usage of those two symbols in
URLs?

In any case it's weird that the URL lexeme doesn't span the same text
as the url_path one, but I'm not sure which one we should consider
wrong.

regards, tom lane