Re: [HACKERS] Challenges preventing us moving to 64 bit transaction id (XID)?

Dear Amit,

Thank you for the attention to this patch.

On Thu, Nov 23, 2017 at 4:39 PM, Amit Kapila <amit(dot)kapila16(at)gmail(dot)com>
wrote:

> On Thu, Jun 22, 2017 at 9:06 PM, Alexander Korotkov
> <a(dot)korotkov(at)postgrespro(dot)ru> wrote:
> > Work on this patch took longer than I expected. It is still in not so
> good
> > shape, but I decided to publish it anyway in order to not stop progress
> in
> > this area.
> > I also tried to split this patch into several. But actually I manage to
> > separate few small pieces, while most of changes are remaining in the
> single
> > big diff.
> > Long story short, patchset is attached.
> >
> > 0001-64bit-guc-relopt-1.patch
> > This patch implements 64 bit GUCs and relation options which are used in
> > further patches.
> >
> > 0002-heap-page-special-1.patch
> > Putting xid and multixact bases into PageHeaderData would take extra 16
> > bytes on index pages too. That would be waste of space for indexes.
> This
> > is why I decided to put bases into special area of heap pages.
> > This patch adds special area for heap pages contaning prune xid and magic
> > number. Magic number is different for regular heap page and sequence
> page.
> >
>
> uint16 pd_pagesize_version;
> - TransactionId pd_prune_xid; /* oldest prunable XID, or zero if none */
> ItemIdData pd_linp[FLEXIBLE_ARRAY_MEMBER]; /* line pointer array */
> } PageHeaderData;
>
> Why have you moved pd_prune_xid from page header?
>

pg_prune_xid makes sense only for heap pages. Once we introduce special
area for heap pages, we can move pg_prune_xid there and save some bytes in
index pages. However, this is an optimization not directly related to
64-bit xids. Idea is that if we anyway change page format, why don't apply
this optimization as well? But if we have any doubts in this, it can be
removed with no problem.

> > 0003-64bit-xid-1.patch
> > It's the major patch. It redefines TransactionID ad 64-bit integer and
> > defines 32-bit ShortTransactionID which is used for t_xmin and t_xmax.
> > Transaction id comparison becomes straight instead of circular. Base
> values
> > for xids and multixact ids are stored in heap page special. SLRUs also
> > became 64-bit and non-circular. To be able to calculate xmin/xmax
> without
> > accessing heap page, base values are copied into HeapTuple.
> Correspondingly
> > HeapTupleHeader(Get|Set)(Xmin|Xmax) becomes just
> > HeapTuple(Get|Set)(Xmin|Xmax) whose require HeapTuple not just
> > HeapTupleHeader. heap_page_prepare_for_xid() is used to ensure that
> given
> > xid fits particular page base. If it doesn't fit then base of page is
> > shifted, that could require single-page freeze. Format for wal is
> changed
> > in order to prevent unaligned access to TransactionId. *_age GUCs and
> > relation options are changed to 64-bit. Forced "autovacuum to prevent
> > wraparound" is removed, but there is still freeze to truncate SLRUs.
> >
>
> It seems there is no README or some detailed explanation of how all
> this works like how the value of pd_xid_base is maintained. I don't
> think there are enough comments in the patch to explain the things. I
> think it will be easier to understand and review the patch if you
> provide some more details either in email or in the patch.
>

OK. I'm going to write README and include it into the patch.

> > 0004-base-values-for-testing-1.patch
> > This patch is used for testing that calculations using 64-bit bases and
> > short 32-bit xid values are correct. It provides initdb options for
> initial
> > xid, multixact id and multixact offset values. Regression tests
> initialize
> > cluster with large (more than 2^32) values.
> >
> > There are a lot of open items, but I would like to notice some of them:
> > * WAL becomes significantly larger due to storage 8 byte xids instead
> of 4
> > byte xids. Probably, its needed to use base approach in WAL too.
> >
>
> Yeah and I think it can impact performance as well. By any chance
> have you run pgbench read-write to see the performance impact of this
> patch?
>

Sure, I'll make some benchmarks on both 32-bit and 64-bit machines.

------
Alexander Korotkov
Postgres Professional: http://www.postgrespro.com
The Russian Postgres Company