Re: Optimizing pglz compressor

On 2013-03-06 11:31:06 -0600, Merlin Moncure wrote:
> On Wed, Mar 6, 2013 at 10:53 AM, Andres Freund <andres(at)2ndquadrant(dot)com> wrote:
> > On 2013-03-06 09:36:19 -0600, Merlin Moncure wrote:
> >> On Wed, Mar 6, 2013 at 8:32 AM, Joachim Wieland <joe(at)mcknight(dot)de> wrote:
> >> > On Tue, Mar 5, 2013 at 8:32 AM, Heikki Linnakangas
> >> > <hlinnakangas(at)vmware(dot)com> wrote:
> >> >> With these tweaks, I was able to make pglz-based delta encoding perform
> >> >> roughly as well as Amit's patch.
> >> >
> >> > Out of curiosity, do we know how pglz compares with other algorithms, e.g. lz4 ?
> >>
> >> This has been a subject of much recent discussion. It compares very
> >> poorly, but installing a new compressor tends to be problematic due to
> >> patent concerns (something which I disagree with but it's there). All
> >> that said, Heikki's proposed changes seem to be low risk and quite
> >> fast.
> >
> > Imo the licensing part is by far the smaller one. The interesting part
> > is making a compatible change to the way toast compression works that
> > supports multiple compression schemes. Afaics nobody has done that work.
> > After that the choice of to-be-integrated compression schemes needs to
> > be discussed, sure.
>
> That wasn't the conversation I remember. lz4 completely smokes pglz
> (Heikki's changes notwithstanding) and is bsd licensed so AFAICS there
> it no reason to support multiple compression technologies except for
> migration purposes (and even if you did want to, a plausible way to do
> that was identified).

Well, we need to permanently support at least two technologies -
otherwise we will break pg_upgrade.
And there is absolutely no reason to believe nothing better than lz4
will come along in the future so just supporting two seems like a bad
idea. And sure, there are rough ideas on how to support different
compression schemes in toast, but nobody has implemented anything
afaics.
It would be possible to reuse what I proposed for indirect toast tuples
for that purpose although I am not sure whether I like using
varatt_external's in multiple types as indicated by
varatt1_be.va_len_1be (renamed to va_type or such) apointing to
different types. Which means that an uncompressible datum would
potentially have multiple representations.

> ...but that's a separate topic for another day. Heikki's proposal
> seems like a win either way.

Yes.

Greetings,

Andres Freund

--
Andres Freund http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Training & Services