Re: libpq compression

On 08.02.2021 22:23, Daniil Zakhlystov wrote:
> Hi everyone,
>
> I’ve been making some experiments with an on-the-fly compression switch lately and have some updates.
>
> ...
> pg_restore of IMDB database test results
>
> Chunked compression with only CopyData or DataRow compression (second approach):
> time:
> real 2m27.947s
> user 0m45.453s
> sys 0m3.113s
> RX bytes diff, human: 1.8837M
> TX bytes diff, human: 1.2810G
>
> Permanent compression:
> time:
> real 2m15.810s
> user 0m42.822s
> sys 0m2.022s
> RX bytes diff, human: 2.3274M
> TX bytes diff, human: 1.2761G
>
> Without compression:
> time:
> real 2m38.245s
> user 0m18.946s
> sys 0m2.443s
> RX bytes diff, human: 5.6117M
> TX bytes diff, human: 3.8227G
>
>
> Also, I’ve run pgbench tests and measured the CPU load. Since chunked compression did not compress any messages
> except for CopyData or DataRow, it demonstrated lower CPU usage compared to the permanent compression, full report with graphs
> is available in the Google doc above.
>
> Pull request with the second approach implemented:
> https://github.com/postgrespro/libpq_compression/pull/7
>
> Also, in this pull request, I’ve made the following changes:
> - extracted the general-purpose streaming compression API into the separate structure (ZStream) so it can be used in other places without tx_func and rx_func,
> maybe the other compression patches can utilize it?
> - made some refactoring of ZpqStream
> - moved the SSL and ZPQ buffered read data checks into separate function pqReadPending
>
> What do you think of the results above? I think that the implemented approach is viable, but maybe I missed something in my tests.

Sorry, but my interpretation of your results is completely different:
permanent compression is faster than chunked compression (2m15 vs. 2m27)
and consumes less CPU (44 vs 48 sec).
Size of RX data is slightly larger - 0.5Mb but TX size is smaller - 5Mb.
So permanent compression is better from all points of view: it is
faster, consumes less CPU and reduces network traffic!

From my point of view your results just prove my original opinion that
possibility to control compression on the fly and use different
compression algorithm for TX/RX data
just complicates implementation and given no significant advantages.

--
Konstantin Knizhnik
Postgres Professional: http://www.postgrespro.com
The Russian Postgres Company