Re: Compression of full-page-writes

(2013/10/22 12:52), Fujii Masao wrote:
> On Tue, Oct 22, 2013 at 12:47 PM, Amit Kapila <amit(dot)kapila16(at)gmail(dot)com> wrote:
>> On Mon, Oct 21, 2013 at 4:40 PM, KONDO Mitsumasa
>> <kondo(dot)mitsumasa(at)lab(dot)ntt(dot)co(dot)jp> wrote:
>>> (2013/10/19 14:58), Amit Kapila wrote:
>>>> On Tue, Oct 15, 2013 at 11:41 AM, KONDO Mitsumasa
>>>> <kondo(dot)mitsumasa(at)lab(dot)ntt(dot)co(dot)jp> wrote:
>>>> In general, my thinking is that we should prefer compression to reduce
>>>> IO (WAL volume), because reducing WAL volume has other benefits as
>>>> well like sending it to subscriber nodes. I think it will help cases
>>>> where due to less n/w bandwidth, the disk allocated for WAL becomes
>>>> full due to high traffic on master and then users need some
>>>> alternative methods to handle such situations.
>>> Do you talk about archiving WAL file?
>>
>> One of the points what I am talking about is sending data over
>> network to subscriber nodes for streaming replication and another is
>> WAL in pg_xlog. Both scenario's get benefited if there is is WAL
>> volume.
>>
>>> It can easy to reduce volume that we
>>> set and add compression command with copy command at archive_command.
>>
>> Okay.
>>
>>>> I think many users would like to use a method which can reduce WAL
>>>> volume and the users which don't find it enough useful in their
>>>> environments due to decrease in TPS or not significant reduction in
>>>> WAL have the option to disable it.
>>> I favor to select compression algorithm for higher performance. If we need
>>> to compress WAL file more, in spite of lessor performance, we can change
>>> archive copy command with high compression algorithm and add documents that
>>> how to compress archive WAL files at archive_command. Does it wrong?
>>
>> No, it is not wrong, but there are scenario's as mentioned above
>> where less WAL volume can be beneficial.
>>
>>> In
>>> actual, many of NoSQLs use snappy for purpose of higher performance.
>>
>> Okay, you can also check the results with snappy algorithm, but don't
>> just rely completely on snappy for this patch, you might want to think
>> of another alternative for this patch.
>
> So, our consensus is to introduce the hooks for FPW compression so that
> users can freely select their own best compression algorithm?
Yes, it will be also good for future improvement. But I think WAL compression for
disaster recovery system should be need in walsender and walreceiver proccess,
and it is propety architecture for DR system. Higher compression ratio with high
CPU usage algorithm in FPW might affect bad for perfomance in master server. If
we can set compression algorithm in walsender and walreciever, performance is
same as before or better, and WAL send performance will be better.

Regards,
--
Mitsumasa KONDO
NTT Open Source Software Center