Re: Add 64-bit XIDs into PostgreSQL 15

On Wed, Jan 5, 2022 at 9:53 PM Alexander Korotkov <aekorotkov(at)gmail(dot)com> wrote:
> I see at least the following major issues/questions in this patch.
> 1) Current code relies on the fact that TransactionId can be
> atomically read from/written to shared memory. With 32-bit systems
> and 64-bit TransactionId, that's not true anymore. Therefore, the
> patch has concurrency issues on 32-bit systems. We need to carefully
> review these issues and provide a fallback for 32-bit systems. I
> suppose nobody is thinking about dropping off 32-bit systems, right?

I think that's right. Not yet, anyway.

> Also, I wonder how critical for us is the overhead for 32-bit systems.
> They are going to become legacy, so overhead isn't so critical, right?

Agreed.

> 2) With this patch we still need to freeze to cut SLRUs. This is
> especially problematic with Multixacts, because systems heavily using
> row-level locks can consume an enormous amount of multixacts. That is
> especially problematic when we have 2x bigger multixacts. We probably
> can provide an alternative implementation for multixact vacuum, which
> doesn't require scanning all the heaps. That is a pretty amount of
> work though. The clog is much smaller and we can cut it more rarely.
> Perhaps, we could tolerate freezing to cut clog, couldn't we?

Right. We can't let any of the SLRUs -- don't forget about stuff like
pg_subtrans, which is a multiple of the size of clog -- grow without
bound, even if it never forces a system shutdown. I'm not sure it's a
good idea to think about introducing new freezing mechanisms at the
same time as we're making other changes, though. Just removing the
possibility of a wraparound shutdown without changing any of the rules
about how and when we freeze would be a significant advancement. Other
changes could be left for future work.

> 3) 2x bigger in-memory representation of TransactionId have overheads.
> In particular, it could mitigate the effect of recent advancements
> from Andres Freund. I'm not exactly sure we should/can do something
> with this. But I think this should be at least carefully analyzed.

Seems fair.

> 4) SP-GiST index stores TransactionId on pages. Could we tolerate
> dropping SP-GiST indexes on a major upgrade? Or do we have to invent
> something smarter?

Probably depends on how much work it is. SP-GiST indexes are not
mainstream, so I think we could at least consider breaking
compatibility, but it doesn't seem like a thing to do lightly.

> 5) 32-bit limitation within the page mentioned upthread by Stephen
> Frost should be also carefully considered. Ideally, we should get rid
> of it, but I don't have particular ideas in this field for now. At
> least, we should make sure we did our best at error reporting and
> monitoring capabilities.

I don't think I understand the thinking here. As long as we retain the
existing limit that the oldest running XID can't be more than 2
billion XIDs in the past, we can't ever need to throw an error. A new
page modification that finds very old XIDs on the page can always
escape trouble by pruning the page and freezing whatever old XIDs
survive pruning.

I would argue that it's smarter not to change the in-memory
representation of XIDs to 64-bit in places like the ProcArray. As you
mention in (4), that might hurt performance. But also, the benefit is
minimal. Nobody is really sad that they can't keep transactions open
forever. They are sad because the system has severe bloat and/or shuts
down entirely. Some kind of change along these lines can fix the
second of those problems, and that's progress.

> I think the realistic goal for PG 15 development cycle would be
> agreement on a roadmap for all the items above (and probably some
> initial implementations).

+1. Trying to rush something through to commit is just going to result
in a bunch of bugs. We need to work through the issues carefully and
take the time to do it well.

--
Robert Haas
EDB: http://www.enterprisedb.com