Re: Ts_rank internals

> I tried to understand how ts_rank works, but I failed. What does Cover
> function do? How does it work? What is the DocRepresentation data
> structure like? I can see the definition of the struct, and the
> get_docrep function to convert to that format, but by reading those I
> can't figure out what the resulting DocRepresentation looks like.
> I wonder if we could get rid of the istrue flag in QueryOperand, and use
> a local BitmapSet variable instead? It seems wrong to have a temporary
> flag that's only used in one function, in a struct that's used everywhere.
It's a play around CDR algorithms (Cover Density Ranking).

Based on paper Clarke et al., “Relevance Ranking for One to Three Term Queries.”
" (http://citeseer.ist.psu.edu/clarke00relevance.html. Sorry, I lost the
article itself, but may be Oleg has it. Simple and short description is placed
at http://www2002.org/CDROM/refereed/643/node7.html.

We change original algorithm to support weight of lexeme, details are on Oleg's
site: http://www.sai.msu.su/~megera/wiki/NewExtentsBasedRanking

Array of DocRepresentation is a representation of document, it contains only
lexemes from both tsvector and tsquery, and lexemes are ordered by position - as
in original doc. Each DocRepresentation has links to corresponding QueryOperand
to optimize query execution while extent search. When we enlarge current
extent for one word then we set istrue flag for corresponding QueryOperand and
execution tsquery from cover becomes very simple task.

It's possible to eliminate istrue flag, but it's needed to implement algorithm
to execute tsquery over continuos part of document, not over whole document.

--
Teodor Sigaev E-mail: teodor(at)sigaev(dot)ru
WWW: http://www.sigaev.ru/