Re: Improving compressibility of WAL files

On Thu, 2009-01-08 at 18:02 -0500, Aidan Van Dyk wrote:
> * Bruce Momjian <bruce(at)momjian(dot)us> [090108 16:43]:
> >
> > The attached patch from Aidan Van Dyk zeros out the end of WAL files to
> > improve their compressibility. (The patch was originally sent to
> > 'general' which explains why it was lost until now.)
> >
> > Would someone please eyeball it?; it is useful for compressing PITR
> > logs even if we find a better solution for replication streaming?
>
> The reason I didn't push it was that people claimed it would chew up to
> much WAL bandwidhh (causing a large commit latency) when the new 0's are
> all written/fsynced at once...
>
> I don't necessarily buy it, because the force_switch is usually either a
> 1) timeed occurance on an otherwise idle time
> 2) user-forced (i.e. forced checkpoint/pg_backup, so your IO is going to
> be hammered anyways...
>
> But that's why I didn't follow up on it...
>
> There's possible a few other ways to do it, such as zero the WAL on
> recycling (but not fsyncing it), and hopefully most of the zero's get
> trickled out by the OS before it comes down to a single 16MB fsync, but
> not many people seemed too enthused about the whole WAL compressablitly
> subject...
>
> But, the way I see things going on -hackers, I must admit, sync-rep (WAL
> streaming) looks like it's a long way off and possibly not even going to
> do what I want, so *I* would really like this wal zero'ing...
>
> If anybody has any specific things with the patch ehty think needs
> chaning, I'll try and accomidate, but I do note that I never
> submitted it for the Commitfest...

won't it still be easier/less intrusive on inline core functionality and
more flexible to just record end-of-valid-wal somewhere and then let the
compressor discard the invalid part when compressing and recreate it
with zeros on decompression ?

-------------------
Hannu