Re: Improving compressibility of WAL files

On Fri, 9 Jan 2009, Simon Riggs wrote:

> Half-filled WAL files were necessary to honour archive_timeout. With
> continuous streaming all WAL files will be 100% full before we switch,
> for most purposes.

The main use case I'm concerned about losing support for is:

1) Two systems connected by a WAN with significant transmit latency
2) The secondary system runs a warm standby aimed at disaster recovery
3) Business requirements want the standby to never be more than (say) 5
minutes behind the primary, presuming the WAN is up
4) WAN traffic is "expensive" (money==bandwidth, one of the two is scarce)

This seems a pretty common scenario in my experience. Right now, this
case is served quite well like this:

-archive_timeout='5 minutes'
-[pglesslog|pg_clearxlogtail] | gzip | rsync

The main concern I have with switching to a more synchronous scheme is
that network efficiency drops as the payload breaks into smaller pieces.
I haven't had enough time to keep up with all the sync rep advances
recently to know for sure if there's a configuration there that's suitable
for this case. If that can be configured to send only in relatively large
chunks, while still never letting things lag too far behind, then I'd
agree completely that the case for any of these WAL cleaner utilities is
dead--presuming said support makes it into the next release.

If that's not available, say because the only useful option sends in very
small pieces, there may still be a need for some utility to fill in for
this particular requirement. Luckily there are many to choose from if it
comes to that.

--
* Greg Smith gsmith(at)gregsmith(dot)com http://www.gregsmith.com Baltimore, MD