Re: bitmaps and correlation

On Sat, Nov 28, 2020 at 5:49 AM Tom Lane <tgl(at)sss(dot)pgh(dot)pa(dot)us> wrote:
>
> Heikki Linnakangas <hlinnaka(at)iki(dot)fi> writes:
> > Other than that, and a quick pgdindent run, this seems ready to me. I'll
> > mark it as Ready for Committer.
>
> I dunno, this seems largely misguided to me.
>
> It's already the case that index correlation is just not the right
> stat for this purpose, since it doesn't give you much of a toehold
> on whether a particular scan is going to be accessing tightly-clumped
> data. For specific kinds of index conditions, such as a range query
> on a btree index, maybe you could draw that conclusion ... but this
> patch isn't paying any attention to the index condition in use.
>
> And then the rules for bitmap AND and OR correlations, if not just
> plucked out of the air, still seem *far* too optimistic. As an
> example, even if my individual indexes are perfectly correlated and
> so a probe would touch only one page, OR'ing ten such probes together
> is likely going to touch ten different pages. But unless I'm
> misreading the patch, it's going to report back an OR correlation
> that corresponds to touching one page.
>
> Even if we assume that the correlation is nonetheless predictive of
> how big a part of the table we'll be examining, I don't see a lot
> of basis for deciding that the equations the patch adds to
> cost_bitmap_heap_scan are the right ones.
>
> I'd have expected this thread to focus a whole lot more on actual
> examples than it has done, so that we could have some confidence
> that these equations have something to do with reality.
>

Status update for a commitfest entry.

The discussion has been inactive since the end of the last CF. It
seems to me that we need some discussion on the point Tom mentioned.
It looks either "Needs Review" or "Ready for Committer" status but
Justin set it to "Waiting on Author" on 2020-12-03 by himself. Are you
working on this, Justin?

Regards,

--
Masahiko Sawada
EDB: https://www.enterprisedb.com/