Re: using Tsearch2 for chemical text

> Rajarshi Guha <rguha(at)indiana(dot)edu> writes:
> > My problem is that the name column contains names of chemicals. Now
> > for many cases this may simply be a number (1674-56-2) and in other
> > cases it may be an alphanumeric string (such as (-)O-acetylcarnitine
> > or 1,2-cis-dihydroxybenzoate). In some cases it is a well-known word
> > (say viagra or calcium chloride or pentathol).
>
> > My question is: will Tsearch2 be able to handle this type of text?
>
> I think you might need to write a custom lexer to divide the strings
> into meaningful units. If there are subsections of these names that
> make sense to search for, then tsearch2 can certainly handle the
> mechanics of that, but I doubt that the standard rules will divide
> these names into lexemes usefully.
>
> regards, tom lane

We have similar problem since Japanese is an agglutinative
language. To solve the problem, we divide Japanese texts into space
separted "words" by using specialized tool, which has huge dictionary
to look for word boundaries. To make things easier, I have written a
simple C function which calls the tool and returns the space separated
texts.

Just for your information.
--
Tatsuo Ishii
SRA OSS, Inc. Japan