Re: strange nested loop row count estimates

Sergey Koposov <skoposov(at)cmu(dot)edu> writes:
> On Thu, 2019-05-02 at 00:36 -0400, Tom Lane wrote:
>> What sort of selectivity estimator have you got attached to that custom
>> operator?

> This is the code, but basically it is just a constant based on the search radius (which is the leftmost float argument of the operator)
> https://github.com/segasai/q3c/blob/361140d4f1f36bf16c9c53721d1c4f03cb4de930/q3c.c#L89

Hm, that query should be paying attention to join selectivity, and
you don't have a join selectivity function.

I think that it applies the restriction selectivity while
estimating the size of the bitmap scan's output. But that's not
what's going to determine the estimated size of the join output.

Too tired to look at this really closely, but I think basically
the inconsistency boils down to the lack of consistency between
your restriction estimator (1e-12) and your join estimator
(which, since you haven't got one, is going to default to
something way larger, possibly 0.5).

regards, tom lane