Re: MaxOffsetNumber for Table AMs

On Fri, 2021-04-30 at 10:50 -0700, Peter Geoghegan wrote:
> I don't know. This conversation is still too abstract for me to be
> able to take a firm position. ISTM that we tend to talk about the
> table AM in excessively abstract terms. It would be a lot easier if
> we
> had clear fixed goals for a small number of additional table AMs.

https://github.com/citusdata/citus/tree/master/src/backend/columnar

My colleagues and I have been working on a "columnar" table AM. It
doesn't currently support indexes, but it would be useful to support
them.

The basic idea is we have "stripes" of ~150000 tuples that are
rearranged and compressed, and stored in an smgr-controlled file that
goes through the buffer cache and uses generic WAL.

To support indexes, we could do our own lookups from a "row number" to
a particular offset where we can find and decompress the stripe that
holds that row number, and then scan forward in the stripe to find the
particular row. This will be terrible for random access, but [*waves
hands*] we will keep state and use a few optimizations so that this is
not terribly slow for clustered access.

Granted, TID lookup on columnar will be much slower than for a heap
(and we can use a CustomScan so that the costs reflect that). But it
will satisfy important use cases:

1. Indexes defined on partition parent tables. Even if the index is
never used for queries, we don't want to throw an error when defining
the partitioned parent index.
2. Unique indexes and exclusion constraints.
3. Clustered index scans can still be reasonably fast.
4. Could be used for UPDATE/DELETE as well.

> More generally, it seems like a good idea to try to make new table
> AMs
> reasonably close to heapam insofar as possible. The reality is that
> everything evolved around heapam, and that that's likely to matter in
> all kinds of ways that nobody fully understands just yet.

Agreed. I think of this as an evolving situation where we take steps
toward a better abstraction.

One (hopefully reasonable) step I'd like to take is a well-specified
TID.

Regards,
Jeff Davis