Re: Optimizing pglz compressor

On 2013-03-06 09:08:10 -0800, Jeff Janes wrote:
> On Wed, Mar 6, 2013 at 8:53 AM, Andres Freund <andres(at)2ndquadrant(dot)com>wrote:
>
> > On 2013-03-06 09:36:19 -0600, Merlin Moncure wrote:
> > > On Wed, Mar 6, 2013 at 8:32 AM, Joachim Wieland <joe(at)mcknight(dot)de> wrote:
> > > > On Tue, Mar 5, 2013 at 8:32 AM, Heikki Linnakangas
> > > > <hlinnakangas(at)vmware(dot)com> wrote:
> > > >> With these tweaks, I was able to make pglz-based delta encoding
> > perform
> > > >> roughly as well as Amit's patch.
> > > >
> > > > Out of curiosity, do we know how pglz compares with other algorithms,
> > e.g. lz4 ?
> > >
> > > This has been a subject of much recent discussion. It compares very
> > > poorly, but installing a new compressor tends to be problematic due to
> > > patent concerns (something which I disagree with but it's there). All
> > > that said, Heikki's proposed changes seem to be low risk and quite
> > > fast.
> >
> > Imo the licensing part is by far the smaller one. The interesting part
> > is making a compatible change to the way toast compression works that
> > supports multiple compression schemes. Afaics nobody has done that work.
> > After that the choice of to-be-integrated compression schemes needs to
> > be discussed, sure.
> >
>
>
> Another thing to consider would be some way of recording an exemplar value
> for each column which is used to seed whatever compression algorithm is
> used. I think there often a lot of redundancy that does not appear within
> any given value, but does appear when viewing all the values of a given
> column. Finding some way to take advantage of that could give a big
> improvement in compression ratio.

But then that value cannot be changed/removed because existing values
depend on it. So either its a one of thing - which means you can only
compute it after the table is filled to some extent - or you need to
have a growing list of such values somewhere (refcounting it would be
hard).
I think the more reasonable route for such a thing would be to try and
get page-level compression working. Which is a pretty hard project, I
admit.

Greetings,

Andres Freund

--
Andres Freund http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Training & Services