Calculating selectivity for the query-planner on ts_vector colums.

Hi

It seems to me that the row estimates on a ts_vector search is a bit on
the low side for terms that is not in th MCV-list in pg_stats:

ftstest=# explain select id from ftstest where ftstest_body_fts @@
to_tsquery('nonexistingterm') order by id limit 10;
QUERY PLAN

---------------------------------------------------------------------------------------------
Limit (cost=221.93..221.95 rows=10 width=4)
-> Sort (cost=221.93..222.01 rows=33 width=4)
Sort Key: id
-> Bitmap Heap Scan on ftstest (cost=154.91..221.22 rows=33
width=4)
Recheck Cond: (ftstest_body_fts @@
to_tsquery('nonexistingterm'::text))
-> Bitmap Index Scan on ftstest_tfs_idx
(cost=0.00..154.90 rows=33 width=0)
Index Cond: (ftstest_body_fts @@
to_tsquery('nonexistingterm'::text))
(7 rows)

Then I have been reading:
http://www.postgresql.org/docs/8.4/static/row-estimation-examples.html
and trying to reproduce the selectivity number for this query:

selectivity = (1 - sum(mvf))/(num_distinct - num_mcv)

num_distinct is around 10m.
ftstest=# SELECT
attname,array_dims(most_common_vals),array_dims(most_common_freqs) FROM
pg_stats WHERE tablename='ftstest' AND
attname='ftstest_body_fts';
attname | array_dims | array_dims
------------------+------------+------------
ftstest_body_fts | [1:2088] | [1:2090]
(1 row)

ftstest=# select tablename,attname,freq from (select tablename,attname,
sum(freq) as freq from (SELECT
tablename,attname,unnest(most_common_freqs) as freq FROM pg_stats) as
foo group by tablename,attname) as foo2 where freq > 1;
tablename | attname | freq
-----------+------------------+---------
ftstest | ftstest_body_fts | 120.967
(1 row)

then the selectivity is
(1-120.967)/(10000000 - 2088) = -.00001199920543409463

Which seem .. well wrong.

The algorithm makes the assumption that if a record is matching one of
the MCV's then it is not in the matching a rare-term. The above
algorithm doesnt give me the 33 rows about, so can anyone shortly
describe the changes for this algorithm when using ts_vectors?

Thanks.

--
Jesper