Re: libpq compression

On 05/15/2018 08:53 AM, Konstantin Knizhnik wrote:
>
>
> On 15.05.2018 13:23, Dmitry Dolgov wrote:
>> > On 30 March 2018 at 14:53, Konstantin Knizhnik
>> <k(dot)knizhnik(at)postgrespro(dot)ru <mailto:k(dot)knizhnik(at)postgrespro(dot)ru>> wrote:
>> > Hi hackers,
>> > One of our customers was managed to improve speed about 10 times by
>> using SSL compression for the system where client and servers are
>> located in different geographical regions
>> > and query results are very large because of JSON columns. Them
>> actually do not need encryption, just compression.
>> > I expect that it is not the only case where compression of libpq
>> protocol can be useful. Please notice that Postgres replication is
>> also using libpq protocol.
>> >
>> > Taken in account that vulnerability was found in SSL compression
>> and so SSLComppression is considered to be deprecated and insecure
>> (http://www.postgresql-archive.org/disable-SSL-compression-td6010072.html)
>> it will be nice to have some alternative mechanism of reducing  libpq
>> traffic.
>> >
>> > I have implemented some prototype implementation of it (patch is
>> attached).
>> > To use zstd compression, Postgres should be configured with
>> --with-zstd. Otherwise compression will use zlib unless it is
>> disabled by --without-zlib option.
>> > I have added compression=on/off parameter to connection string and
>> -Z option to psql and pgbench utilities.
>>
>> I'm a bit confused why there was no reply to this. I mean, it wasn't
>> sent on
>> 1st April, the patch still can be applied on top of the master branch
>> and looks
>> like it even works.
>>
>> I assume the main concern her is that it's implemented in a rather not
>> extensible way. Also, if I understand correctly, it compresses the
>> data stream
>> in both direction server <-> client, not sure if there is any value in
>> compressing what a client sends to a server. But still I'm wondering
>> why it
>> didn't start at least a discussion about how it can be implemented.
>> Do I miss
>> something?
>
> Implementation of libpq compression will be included in next release
> of PgProEE.
> Looks like community is not so interested in this patch. Frankly
> speaking I do not understand why.
> Compression of libpq traffic can significantly increase speed of:
> 1. COPY
> 2. Replication (both streaming and logical)
> 3. Queries returning large results sets (for example JSON) through
> slow connections.
>
> It is possible to compress libpq traffic using SSL compression. But
> SSL compression is unsafe and deteriorated feature.
>
> Yes, this patch is not extensible: it can use either zlib either zstd.
> Unfortunately internal Postgres compression pglz doesn't provide
> streaming API.
> May be it is good idea to combine it with Ildus patch (custom
> compression methods): https://commitfest.postgresql.org/18/1294/
> In this case it will be possible to use any custom compression
> algorithm. But we need to design and implement streaming API for pglz
> and other compressors.
>
>

I'm sure there is plenty of interest in this. However, you guys need to
understand where we are in the development cycle. We're trying to wrap
up Postgres 11, which was feature frozen before this patch ever landed.
So it's material for Postgres 12. That means it will probably need to
wait a little while before it gets attention. It doesn't mean nobody is
interested.

I disagree with Dmitry about compressing in both directions - I can
think of plenty of good cases where we would want to compress traffic
from the client.

cheers

andrew

--
Andrew Dunstan https://www.2ndQuadrant.com
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services