Re: XLog size reductions: smaller XLRec block header for PG17

On Tue, 19 Sept 2023 at 01:03, Andres Freund <andres(at)anarazel(dot)de> wrote:
>
> Hi,
>
> On 2023-05-18 19:22:26 +0300, Heikki Linnakangas wrote:
> > On 18/05/2023 17:59, Matthias van de Meent wrote:
> > > It changes the block IDs used to fit in 6 bits, using the upper 2 bits
> > > of the block_id field to store how much data is contained in the
> > > record (0, <=UINT8_MAX, or <=UINT16_MAX bytes).
> >
> > Perhaps we should introduce a few generic inline functions to do varint
> > encoding. That could be useful in many places, while this scheme is very
> > tailored for XLogRecordBlockHeader.

This scheme is reused later for the XLogRecord xl_tot_len field over
at [0], and FWIW is thus being reused. Sure, it's tailored to this WAL
use case, but IMO we're getting good value from it. We don't use
protobuf or JSON for WAL, we use our own serialization format. Having
some specialized encoding/decoding in that format for certain fields
is IMO quite acceptable.

> Yes - I proposed that and wrote an implementation of reasonably efficient
> varint encoding. Here's my prototype:
> https://postgr.es/m/20221004234952.anrguppx5owewb6n%40awork3.anarazel.de

As I mentioned on that thread, that prototype has a significant
probability of doing nothing to improve WAL size, or even increasing
the WAL size for installations which consume a lot of OIDs.

> I think it's a bad tradeoff to write lots of custom varint encodings, just to
> eek out a bit more space savings.

This is only a single "custom" varint encoding though, if you can even
call it that. It makes a field's size depend on flags set in another
byte, which is not that much different from the existing use of
XLR_BLOCK_ID_DATA_[LONG, SHORT].

> The increase in code complexity IMO makes it a bad tradeoff.

Pardon me for asking, but what would you consider to be a good
tradeoff then? I think the code relating to the WAL storage format is
about as simple as you can get it within the feature set it provides
and the size of the resulting records. While I think there is still
much to gain w.r.t. WAL record size, I don't think we can get much of
those improvements without adding at least some amount of complexity,
something I think to be true for most components in PostgreSQL.

So, except for redesigning significant parts of the public WAL APIs,
are we just going to ignore any potential improvements because they
"increase code complexity"?

Kind regards,

Matthias van de Meent
Neon (https://neon.tech)

[0] https://commitfest.postgresql.org/43/4386/