Re: MaxOffsetNumber for Table AMs

On Mon, 2021-05-03 at 15:07 -0700, Peter Geoghegan wrote:
> Sure, but it either makes sense for the columnar table AM to support
> bitmap scans (or some analogous type of scan that works only slightly
> differently) or it doesn't. It's not at all clear which it is right
> now.

It makes sense for my columnar table AM -- there's TID locality.

> If it makes sense then it will of course be necessary to describe
> what
> "bitmap scan" actually means with the columnar storage table AM (plus
> you'll still need to make some in-core changes to places like
> tidbitmap.c). OTOH if it doesn't make sense then that's that -- it's
> going to be a bit annoying in the partitioning scenario you describe,
> but some things are bound to be *inherently* impossible, so it can't
> be
> helped.

I don't see why in-core changes are a strict requirement. It doesn't
make too much difference if a lossy TID doesn't correspond exactly to
the columnar layout -- it should be fine as long as there's locality,
right?

> It seems senseless to *require* table AMs to support something like a
> bitmap scan.

I am not yet convinced that it's "senseless", but it is optional and
there's probably a reason that it's not required.

We still need to address the fact that two features have had a minor
collision: indexes on a partitioned table and table AMs that don't
necessarily support all index types. It's not good to just throw an
error, because we could be forcing the user to manually manage the
indexes on hundreds of partitions just because some tables have a
different AM and it doesn't support the index type.

We probably want to do something about that, but as far as I can tell,
it's not a problem for columnar right now.

> I don't think it's a coincidence that GIN is the index AM
> that looks like it presents at least 2 problems for the columnar
> table
> AM. To me this suggests that this will need a much higher level
> discussion.

One problem is that ginpostinglist.c restricts the use of offset
numbers higher than MaxOffsetNumber - 1. At best, that's a confusing
and unnecessary off-by-one error that we happen to be stuck with
because it affects the on-disk format. Now that I'm past that
particular confusion, I can live with a workaround until we do
something better.

What is the other problem with GIN?

Regards,
Jeff Davis