Re: [PATCHES] Post-special page storage TDE support

From: David Christensen <david(dot)christensen(at)crunchydata(dot)com>
To: Matthias van de Meent <boekewurm+postgres(at)gmail(dot)com>
Cc: PostgreSQL-development <pgsql-hackers(at)postgresql(dot)org>, Stephen Frost <sfrost(at)snowman(dot)net>
Subject: Re: [PATCHES] Post-special page storage TDE support
Date: 2022-11-08 16:17:56
Message-ID: CAOxo6XKPWaPP+VaETgQiHbR51Tq5HAdWs6-hE6S9+WLi1U4qPA@mail.gmail.com
Views: Raw Message | Whole Thread | Download mbox | Resend email
Thread:
Lists: pgsql-hackers

Per some offline discussion with Stephen and incorporating some of the
feedback I've gotten I'm including the following changes/revisions:

1. Change the signature of any macros that rely on a dynamic component
to look like a function so you can more easily determine in-code
whether something is truly a constant/compile time calculation or a
runtime one.

2. We use a new page flag for whether "extended page features" are
enabled on the given page. If this is set then we look for the 1-byte
trailer with the bitflag of the number of features. We allow space for
7 page features and are reserving the final hi bit for future
use/change of interpretation to accommodate more.

3. Consolidate the extended checksums into a 56-bit checksum that
immediately precedes the 1-byte flag. Choice of 64-bit checksum is
arbitrary just based on some MIT-licensed code I found, so just
considering this proof of concept, not necessarily promoting that
specific calculation. (I think I included some additional checksum
variants from the earlier revision for ease of testing various
approaches.)

4. Ensure the whole area is MAXALIGN and fixed a few bugs that were
pointed out in this thread.

Patches are:

1. make select_views stable, prerequisite for anything that is messing
with tuples on page sizes

2. add reserved_page_size handling and rework existing code to account
for this additional space usage

3. main PageFeatures-related code; introduce that abstraction layer,
along with the trailing byte on the page with the enabled features for
this specific page. We also add an additional param to PageInit()
with the page features active on this page; currently all call sites
are using the cluster-wide cluster_page_features as the parameter, so
all pages share what is stored in the control file based on initdb
options. However, routines which query page features look on the
actual page itself, so in fact we are able to piecemeal at the
page/relation level if we so desire, or turn off for specific types of
pages, say. Also includes the additional pd_flags bit to enable that
interpretation.

4. Actual extended checksums PageFeature. Rather than two separate
implementations as in the previous patch series, we are using 56 bits
of a 64-bit checksum, stored as the high 7 bytes of the final 8 in the
page where this is enabled.

5. wasted_space PageFeature just to demo multiple features in play.

Thanks,

David

Attachment Content-Type Size
v2-0002-Add-reserved_page_space-to-Page-structure.patch application/octet-stream 65.6 KB
v2-0005-A-second-page-feature-just-to-allocate-more-space.patch application/octet-stream 4.4 KB
v2-0004-Add-extended-page-checksums-feature.patch application/octet-stream 36.7 KB
v2-0003-Add-pagefeat-plus-new-page-status-bit-and-page-fe.patch application/octet-stream 27.8 KB
v2-0001-Make-the-output-of-select_views-test-stable.patch application/octet-stream 218.4 KB

In response to

Responses

Browse pgsql-hackers by date

  From Date Subject
Next Message Tom Lane 2022-11-08 16:28:08 Non-emergency patch for bug #17679
Previous Message Mark Dilger 2022-11-08 14:24:13 Re: Tables not getting vacuumed in postgres