Re: libpq compression

On 05.11.2020 22:22, Peter Eisentraut wrote:
> On 2020-11-02 20:50, Andres Freund wrote:
>> On 2020-10-31 22:25:36 +0500, Andrey Borodin wrote:
>>> But the price of compression is 1 cpu for 500MB/s (zstd). With a
>>> 20Gbps network adapters cost of recompressing all traffic is at most
>>> ~4 cores.
>>
>> It's not quite that simple, because presumably each connection is going
>> to be handled by one core at a time in the pooler. So it's easy to slow
>> down peak throughput if you also have to deal with TLS etc.
>
> Also, current deployments of connection poolers use rather small
> machine sizes.  Telling users you need 4 more cores per instance now
> to decompress and recompress all the traffic doesn't seem very
> attractive. Also, it's not unheard of to have more than one layer of
> connection pooling.  With that, this whole design sounds a bit like a
> heat-generation system. ;-)

Compression will be mostly useful for:
1. Replication protocol
2. COPY command
3. OLAP queries returning large result sets
4. Queries returning BLOBs/JSON

It seems to be not so good idea to switch on compression for all
connections.
And cases describe above are usually affect only small number of backends.

Also please notice, that compression may significantly reduce size of
transferred data. An copying data between multiple levels of network
protocol
also consumes significant amount of CPU. Also number of system calls can
be proportional to the size of transferred data and in many cases
now performances if servers is limited by number of system calls, rather
than by network throughput/latency and other factors.

So I accept your arguments, but still think that picture is not so
straightforward.
This is why it is very interesting to me to know results of using
compression with Odyssey in real production environment.