Compression of bigger WAL records

From: "Andrey M(dot) Borodin" <x4mmm(at)yandex-team(dot)ru>
To: pgsql-hackers <pgsql-hackers(at)postgresql(dot)org>
Subject: Compression of bigger WAL records
Date: 2025-01-12 12:43:25
Message-ID: 4DC38068-976E-4A84-8EE6-4EFACBBD927A@yandex-team.ru
Views: Raw Message | Whole Thread | Download mbox | Resend email
Thread:
Lists: pgsql-hackers

Hi hackers!

I propose a slight change to WAL compression: compress body of big records, if it's bigger than some threshold.

===Rationale===
0. Better compression ratio for full page images when pages are compressed together.

Consider following test:

set wal_compression to 'zstd';
create table a as select random() from generate_series(1,1e7);
create index on a(random ); -- warmup to avoid FPI for hint on the heap
select pg_stat_reset_shared('wal'); create index on a(random ); select pg_size_pretty(wal_bytes) from pg_stat_wal;

B-tree index will emit 97Mb of WAL instead of 125Mb when FPIs are compressed independently.

1. Compression of big records, that are not FPI. E.g. 2-pc records might be big enough to cross a threshold.

2. This might be a path to full WAL compression. In future I plan to propose a compression context: retaining compression dictionary between records. Obviously, the context cannot cross checkpoint borders. And a pool of contexts would be needed to fully utilize efficiency of compression codecs. Anyway - it's too early to theorize.

===Propotype===
I attach a prototype patch. It is functional, but some world tests fail. Probably, because they expect to generate more WAL without putting too much of entropy. Or, perhaps, I missed some bugs. In present version WAL_DEBUG does not indicate any problems. But a lot of quality assurance and commenting work is needed. It's a prototype.

To indicate that WAL record is compressed I use a bit in record->xl_info (XLR_COMPRESSED == 0x04). I found no places that use this bit...
If the record is compressed, record header is continued with information about compression: codec byte and uint32 of uncompressed xl_tot_len.

Currently, compression is done on StringInfo buffers, that are expanded before actual WALInsert() happens. If palloc() is needed during critical section, the compression is canceled. I do not like memory accounting before WALInsert, probably, something clever can be done about it.

WAL_DEBUG and wal_compression are enabled for debugging purposes. Of course, I do not propose to turn them on by default.

What do you think? Does this approach seem viable?

Best regards, Andrey Borodin.

Attachment Content-Type Size
v0-0001-Compress-big-WAL-records.patch application/octet-stream 34.3 KB

Responses

Browse pgsql-hackers by date

  From Date Subject
Next Message Dean Rasheed 2025-01-12 14:10:29 Re: Missing quotes when deparsing XMLTABLE() and SQL/JSON query functions
Previous Message Masahiko Sawada 2025-01-12 09:34:56 Re: Parallel heap vacuum