Re: Compression of bigger WAL records

On Tue, 21 Jan 2025 at 23:24, "Andrey M. Borodin" <x4mmm(at)yandex-team(dot)ru> wrote:
>> On 12 Jan 2025, at 17:43, Andrey M. Borodin <x4mmm(at)yandex-team(dot)ru> wrote:
>>
>> I attach a prototype patch.
>
> Here's v2, now it passes all the tests with wal_debug.
>
> Some stats. On this test
>
> create table a as select random() from generate_series(1,1e7);
> select pg_stat_reset_shared('wal'); create index on a(random ); select pg_size_pretty(wal_bytes) from pg_stat_wal;
> set wal_compression to 'lz4';
> select pg_stat_reset_shared('wal'); create index on a(random ); select pg_size_pretty(wal_bytes) from pg_stat_wal;
> set wal_compression to 'pglz';
> select pg_stat_reset_shared('wal'); create index on a(random ); select pg_size_pretty(wal_bytes) from pg_stat_wal;
> set wal_compression to 'zstd';
> select pg_stat_reset_shared('wal'); create index on a(random ); select pg_size_pretty(wal_bytes) from pg_stat_wal;
>
> I observe WAL size of the index:
> method HEAD patched
> pglz 193 MB 193 MB
> lz4 160 MB 132 MB
> zstd 125 MB 97 MB
>
> So, for lz4 and zstd this seems to be a significant reduction.
>
> I'm planning to work on improving the patch quality.
>
> Thanks!
>

Hi, Andrey Borodin

I find this feature interesting; however, it cannot be applied to the current
master (b35434b134b) due to commit 32a18cc0a73.

Applying: Compress big WAL records
.git/rebase-apply/patch:83: trailing whitespace.

.git/rebase-apply/patch:90: trailing whitespace.

.git/rebase-apply/patch:315: trailing whitespace.

.git/rebase-apply/patch:780: trailing whitespace.
else
error: contrib/pg_walinspect/pg_walinspect.c: does not match index
error: src/backend/access/rmgrdesc/xlogdesc.c: does not match index
error: src/backend/access/transam/xlog.c: does not match index
error: src/backend/access/transam/xloginsert.c: does not match index
error: src/backend/access/transam/xlogreader.c: does not match index
error: src/backend/utils/misc/guc_tables.c: does not match index
error: src/backend/utils/misc/postgresql.conf.sample: does not match index
error: src/include/access/xlog.h: does not match index
error: src/include/access/xloginsert.h: does not match index
error: src/include/access/xlogreader.h: does not match index
error: src/include/access/xlogrecord.h: does not match index
error: src/include/pg_config_manual.h: does not match index
error: src/test/recovery/t/026_overwrite_contrecord.pl: does not match index
error: patch failed: src/test/recovery/t/039_end_of_wal.pl:81
error: src/test/recovery/t/039_end_of_wal.pl: patch does not apply
Patch failed at 0001 Compress big WAL records
hint: Use 'git am --show-current-patch=diff' to see the failed patch
When you have resolved this problem, run "git am --continue".
If you prefer to skip this patch, run "git am --skip" instead.
To restore the original branch and stop patching, run "git am --abort".

I see the patch compresses the WAL record according to the wal_compression,
IIRC the wal_compression is only used for FPI, right? Maybe we should update
the description of this parameter.

I see that the wal_compression_threshold defaults to 512. I wonder if you
chose this value based on testing or randomly.

--
Regrads,
Japin Li