Re: libpq compression

> 26 февр. 2021 г., в 02:18, Daniil Zakhlystov <usernamedt(at)yandex-team(dot)ru> написал(а):
>
> Yes, there is still no possibility for compressed traffic pass-through for poolers,
> but do we actually need it?
> I don’t see any solution except starting a new compression context for
> each message in order to make it work.
> Do we actually need to hurt the compression ratio for this specific use case?
>
> Actually, there is an ugly hack - we may force-reset the compression context by sending the
> SetCompressionMethod (which will reset the compression algorithm & context) after each CompressedData message.
>
> This should allow interpreting of each CompressedData message on its own but will add overhead and hurt the compression ratio.

As a maintainer of a connection pooler, I don't think compressing individual messages on their own compression context adds any value for that particular pooler.
Network has two different costs:
1. Cross-datacenter traffic is costly and needs to be compressed. Better compression ratios are preferable to CPU savings.
2. Intra-datacenter traffic is cheap and compression is not that crucial. In my opinion on most loaded installations, the pooler must be on the same host as DB. Compression can be just off.

I don't think we should craft one more CPU\Network tradeoff point by our own hands. Enough tradeoff points are implemented by lz4\zstd\whatever.

CPU usage to compress data is neglectable small compared to CPU overhead to produce data traffic.

The only real feature of compressing individual messages is better CRIME mitigation. E.i. not compressing Auth packets and some parameters of the query. This feature does not depend on resetting compression context. And resetting compression context often does not help to mitigate CRIME.

Thanks!

Best regards, Andrey Borodin.