Re: Different compression methods for FPI

> 25 мая 2021 г., в 12:26, Michael Paquier <michael(at)paquier(dot)xyz> написал(а):
>
> On Mon, May 24, 2021 at 11:44:45PM -0500, Justin Pryzby wrote:
>> The goal is to support 2+ "methods" (including "none"), which takes 4 bits, so
>> may as well support 3 methods.
>>
>> - uncompressed
>> - pglz
>> - lz4
>> - zlib or zstd or ??
>
> Let's make a proper study of all that and make a choice, the only
> thing I am rather sure of is that pglz is bad compared to all the
> others. There is no point to argue as long as we don't know if any of
> those candidates are suited for the job.

There's a lot of open studies like [0,1].
In short, Lz4 is fastest codec. Zstd gives better compression at cost of more CPU usage[2]. Zlib is not tied to Facebook, however, it's slower and have smaller compression ratio than Zstd. Zstd is actively developed.

There is Google's Brotli codec. It is comparable to Zstd.

What kind of deeper study do we want to do?
Would it make sense to run our own benchmarks?
Or, perhaps, review other codecs?

In my opinion, anything that is sent over network or written to block storage deserves Zstd-5 compression. But milage may vary.

Thanks!

Best regards, Andrey Borodin.

[0] https://indico.cern.ch/event/695984/contributions/2872933/attachments/1590457/2516802/ZSTD_and_ZLIB_Updates_-_January_20186.pdf
[1] https://facebook.github.io/zstd/
[2] Zstd gives significantly better compression at cost of little more CPU usage. But they both have points on Pareto frontier.