Re: Faster compression, again

On Thu, Mar 15, 2012 at 3:14 PM, Simon Riggs <simon(at)2ndquadrant(dot)com> wrote:
> On Wed, Mar 14, 2012 at 6:06 PM, Daniel Farina <daniel(at)heroku(dot)com> wrote:
>
>> If we're curious how it affects replication
>> traffic, I could probably gather statistics on LZO-compressed WAL
>> traffic, of which we have a pretty huge amount captured.
>
> What's the compression like for shorter chunks of data? Is it worth
> considering using this for the libpq copy protocol and therefore
> streaming replication also?

The overhead is between 1 and 5 bytes that reserve the high bit as a
continuation bit (so one byte for small data), and then straight into
data. So I think it could be applied for most payloads that are a few
bytes wide. Presumably that could be lifted, but the format
description only allows for 2**32 - 1 for the uncompressed size.

I'd really like to find a way to layer both message-oblivious and
message-aware transport under FEBE with both backend and frontend
support without committing the project to new code for-ever-and-ever.
I guess I could investigate it in brief now, unless you've already
thought about/done some work in that area.

--
fdr