Re: strange nested loop row count estimates

On Thu, 2019-05-02 at 01:05 -0400, Tom Lane wrote:
> Sergey Koposov <skoposov(at)cmu(dot)edu> writes:
> >
> > On Thu, 2019-05-02 at 00:36 -0400, Tom Lane wrote:
> > >
> > > What sort of selectivity estimator have you got attached to that custom
> > > operator?
> >
> > This is the code, but basically it is just a constant based on the search radius (which is the leftmost float argument of the operator)
> > https://github.com/segasai/q3c/blob/361140d4f1f36bf16c9c53721d1c4f03cb4de930/q3c.c#L89
> Hm, that query should be paying attention to join selectivity, and
> you don't have a join selectivity function.
>
> I think that it applies the restriction selectivity while
> estimating the size of the bitmap scan's output.  But that's not
> what's going to determine the estimated size of the join output.
>
> Too tired to look at this really closely, but I think basically
> the inconsistency boils down to the lack of consistency between
> your restriction estimator (1e-12) and your join estimator
> (which, since you haven't got one, is going to default to
> something way larger, possibly 0.5).

Thanks very much checking, Tom!  
Adding the join selectivity estimator fixed the problem. 
I think I initially tried it, but it wasn't clear whether it was called at all or not.
Plus I was confused by the fact that the bitmap scan prediction showed 1 row, so it looked like the selectivity worked. 

        Sergey