Re: Compression of full-page-writes

On Fri, Oct 11, 2013 at 5:05 AM, Andres Freund <andres(at)2ndquadrant(dot)com> wrote:
> Hi,
> On 2013-10-11 03:44:01 +0900, Fujii Masao wrote:
>> I'm afraid that the patch has only limited effects in WAL reduction and
>> performance improvement unless the database contains highly-compressible
>> data like large blank characters column. It really depends on the contents
>> of the database. So, obviously FPW compression should not be the default.
>> Maybe we can treat it as just tuning knob.
>
>
> Have you tried using lz4 (or snappy) instead of pglz? There's a patch
> adding it to pg in
> http://archives.postgresql.org/message-id/20130621000900.GA12425%40alap2.anarazel.de
>
> If this really is only a benefit in scenarios with lots of such data, I
> have to say I have my doubts about the benefits of the patch.

I think it will be difficult to prove by using any compression
algorithm, that it compresses in most of the scenario's.
In many cases it can so happen that the WAL will also not be reduced
and tps can also come down if the data is non-compressible, because
any compression algorithm will have to try to compress the data and it
will burn some cpu for that, which inturn will reduce tps.

As this patch is giving a knob to user to turn compression on/off, so
users can decide if they want such benefit.
Now some users can say that they have no idea, how or what kind of
data will be there in their databases, so such kind of users should
not use this option, but on the other side some users know that they
have similar pattern of data, so they can get benefit out of such
optimisations. For example in telecom industry, i have seen that they
have lot of data as CDR's (call data records) in their HLR databases
for which the data records will be different but of same pattern.

Being said above, I think both this patch and my patch "WAL reduction
for Update" (https://commitfest.postgresql.org/action/patch_view?id=1209)
are using same technique for WAL compression and can lead to similar
consequences in different ways.
So I suggest to have unified method to enable WAL Compression for both
the patches.

With Regards,
Amit Kapila.
EnterpriseDB: http://www.enterprisedb.com