Re: Quick-and-dirty compression for WAL backup blocks

> Date: Tue, 31 May 2005 16:26:20 -0400
> From: Tom Lane <tgl(at)sss(dot)pgh(dot)pa(dot)us>
> To: pgsql-hackers(at)postgreSQL(dot)org
> Subject: Quick-and-dirty compression for WAL backup blocks
> Message-ID: <23210(dot)1117571180(at)sss(dot)pgh(dot)pa(dot)us>

(cut)

> ... make the WAL writing logic aware of the layout
> of buffer pages --- specifically, to know that our pages generally
> contain an uninteresting "hole" in the middle, and not write the hole.
> Optimistically this might reduce the WAL data volume by something
> approaching 50%; though pessimistically (if most pages are near full)
> it wouldn't help much.

> A more concrete version of this is: examine the page to see if the
> pd_lower field is between SizeOfPageHeaderData and BLCKSZ, and if so
> whether there is a run of consecutive zero bytes beginning at the
> pd_lower position. Omit any such bytes from what is written to WAL.
> (This definition ensures that nothing goes wrong if the page does not
> follow the normal page layout conventions: the transformation is
> lossless no matter what, since we can always reconstruct the exact page
> contents.) The overhead needed is only 2 bytes to show the number of
bytes removed.

I can't say I'm familiar enough with the exact format of the XLog records
(wasn't sure where to find this), but from what you're saying (and by
definition I suppose) I'm guessing it is just a complete copy of the new
page just before it gets written to the heap?

I also noticed your comment above that mentioned that compression would be
less effective as the pages became more full. Would changing the loading
factor of database pages have an effect here, as I would have thought that
the WAL would be fsync'd more aggressively than the heap?

> A run-length compressor would be reasonably quick but I think that the
> omit-the-middle-
> hole approach gets most of the possible win with even less work.

I can't think that a RLE scheme would be much more expensive than a 'count
the hole' approach with more benefit, so I wouldn't like to discount this
straight away...

If you do manage to go ahead with the code, I'd be very interested to see
some comparisons in bytes written to XLog for old and new approaches for
some inserts/updates. Perhaps we could ask Mark to run another TPC benchmark
at OSDL when this and the CRC changes have been completed.

Kind regards,

Mark.

------------------------
WebBased Ltd
South West Technology Centre
Tamar Science Park
Plymouth
PL6 8BT

T: +44 (0)1752 797131
F: +44 (0)1752 791023
W: http://www.webbased.co.uk