Re: MaxOffsetNumber for Table AMs

On Wed, May 5, 2021 at 7:27 AM Robert Haas <robertmhaas(at)gmail(dot)com> wrote:
> It seems to me that we're doing a lot of disagreeing given that, as I
> see it, there are only relatively minor differences between the
> positions of the various people here.

I'm being very vocal here because I'm concerned that we're going about
generalizing TIDs in the wrong way. To me it feels like there is a
loss of perspective about what really matters. There just isn't that
many table AM TID designs that could ever work, and even among those
schemes that could ever work there is a pretty clear hierarchy. This
blue sky thinking about generalizing TIDs 2 years in seems *weird* to
me.

Nobody is obligated to reassure me. I felt that this had to be said.

> Andres and I are, I think,
> relatively convinced that variable-width TIDs would let us do things
> that aren't otherwise possible, and that those things are potentially
> useful and I would even venture to say cool. I don't believe you
> disagree with that, but you think it's going to be too much work to
> implement. Fair enough; anyone can try it who is interested and see
> how far they get. Anyone who thinks it's going to be impossibly hard
> probably will prefer not to try, and that's OK too.

I think that's accurate. But it's easy to not disagree with the idea
that variable-width TIDs might lead to something interesting. Talk is
cheap.

No other database system has something like indirect indexes. They
have clustered indexes, but that's rather different. I think that
indirect indexes were a design that was concerned about the issue of
write amplification from non-HOT updates. But do I even remember the
details correctly? We're talking about indirect indexes as if that was
an idea whose high level user-visible goals were clear, but I don't
even think that that much is true. This kind of thing concerns me. It
very much feels like failing to see the forest for the trees.

> But if we take that off the table, what about a less-ambitious
> generalization of the TID mechanism? I can't really see anyone putting
> up a serious argument against allowing all 48 bits of space available
> in the existing TID format to be used which, as Jeff points out, is
> not currently the case. So anyone who wants to try to write that patch
> is free to do so. I don't have a clear idea how to make that work, to
> be honest, but my limited supply of ideas need not prevent anyone else
> from trying theirs.

I agree that we should focus on what we can agree on. It seems as if
we all more or less agree on this much.

> There might be some slight disagreement about whether it's useful to
> generalize TIDs from a 48-bit address space to a 64-bit address space
> without making it fully general. Like Andres, I am unconvinced that's
> meaningfully easier, and I am convinced that it's meaningfully less
> good, but other people can disagree and that's fine. I'm perfectly
> willing to change my opinion if somebody shows up with a patch that
> demonstrates the value of this approach.

It's going to be hard if not impossible to provide empirical evidence
for the proposition that 64-bit wide TIDs (alongside 48-bit TIDs) are
the way to go. Same with any other scheme. We're talking way too much
about TIDs themselves and way too little about table AM use cases, the
way the data structures might work in new table AMs, and so on.

> The main point here is one that I think you made a few emails back:
> the limitations of the current system are defined by what will
> actually work with the code as it exists today, not some mailing list
> discussion.

Right.

> It's too early for the project to commit to stability in
> this area; we have not managed to get a single AM apart from heapam
> into core, and that situation doesn't appear likely to change in the
> near future.

I would be happy if we could commit to committing to stability. I
really don't think that it's *that* hard to move significantly closer
to a design that describes just how close to heapam a table AM should
be. It doesn't commit the table AM to all that many details.

> If and when we have say 5 of those we can probably
> articulate some intelligent ideas about what we think the patterns
> that need to hold for future AMs are, but it's reckless to extrapolate
> from 1 working example, and right now that's all we have.

I don't think that there will be 5 table AMs that are credible to
users at any point in the future. In any case there only needs to be 1
or 2 good ones for the table AM to have been a resounding success.

--
Peter Geoghegan