Re: Add 64-bit XIDs into PostgreSQL 15

Hi!

On Thu, Jan 6, 2022 at 3:02 AM Bruce Momjian <bruce(at)momjian(dot)us> wrote:
>
> On Thu, Dec 30, 2021 at 03:15:16PM +0300, Maxim Orlov wrote:
> > PFA updated working patch v6 for PG15 development cycle.
> > It is based on a patch by Alexander Korotkov version 5 [5] with a few fixes,
> > refactoring and was rebased to PG15.
> >
> > Main changes:
> > - Change TransactionId to 64bit
> > - Disk tuple format (HeapTupleHeader) is unchanged: xmin and xmax
> > remains 32bit
> > -- 32bit xid is named ShortTransactionId now.
> > -- Exception: see "double xmax" format below.
> > - Heap page format is changed to contain xid and multixact base value,
> > tuple's xmin and xmax are offsets from.
> > -- xid_base and multi_base are stored as a page special data. PageHeader
> > remains unmodified.
> > -- If after upgrade page has no free space for special data, tuples are
> > converted to "double xmax" format: xmin became virtual
> > FrozenTransactionId, xmax occupies the whole 64bit.
> > Page converted to new format when vacuum frees enough space.
>
> I think it is a great idea to allow the 64-XID to span the 32-bit xmin
> and xmax fields when needed. It would be nice if we can get focus on
> this feature so we are sure it gets into PG 15. Can we add this patch
> incrementally so people can more easily analyze it?

I see at least the following major issues/questions in this patch.
1) Current code relies on the fact that TransactionId can be
atomically read from/written to shared memory. With 32-bit systems
and 64-bit TransactionId, that's not true anymore. Therefore, the
patch has concurrency issues on 32-bit systems. We need to carefully
review these issues and provide a fallback for 32-bit systems. I
suppose nobody is thinking about dropping off 32-bit systems, right?
Also, I wonder how critical for us is the overhead for 32-bit systems.
They are going to become legacy, so overhead isn't so critical, right?
2) With this patch we still need to freeze to cut SLRUs. This is
especially problematic with Multixacts, because systems heavily using
row-level locks can consume an enormous amount of multixacts. That is
especially problematic when we have 2x bigger multixacts. We probably
can provide an alternative implementation for multixact vacuum, which
doesn't require scanning all the heaps. That is a pretty amount of
work though. The clog is much smaller and we can cut it more rarely.
Perhaps, we could tolerate freezing to cut clog, couldn't we?
3) 2x bigger in-memory representation of TransactionId have overheads.
In particular, it could mitigate the effect of recent advancements
from Andres Freund. I'm not exactly sure we should/can do something
with this. But I think this should be at least carefully analyzed.
4) SP-GiST index stores TransactionId on pages. Could we tolerate
dropping SP-GiST indexes on a major upgrade? Or do we have to invent
something smarter?
5) 32-bit limitation within the page mentioned upthread by Stephen
Frost should be also carefully considered. Ideally, we should get rid
of it, but I don't have particular ideas in this field for now. At
least, we should make sure we did our best at error reporting and
monitoring capabilities.

I think the realistic goal for PG 15 development cycle would be
agreement on a roadmap for all the items above (and probably some
initial implementations).

------
Regards,
Alexander Korotkov