Re: Wire protocol compression

On 21 April 2016 at 22:07, Tom Lane <tgl(at)sss(dot)pgh(dot)pa(dot)us> wrote:

>
> Yeah. I think this should definitely be in the list of things we want
> to add when we do the fabled 4.0 protocol revision (and, indeed, it's
> been in the above-cited list for awhile). Whether we've yet gotten to
> the point of having critical mass for a revision ... meh, I doubt it.
>
>
Well, the critical mass is determined as much as anything by who's willing
to put the large amount of work into implementing the protocol rev, testing
clients, verifying performance, etc etc etc. So far I don't think anyone's
leaping to volunteer.

We could use various hacks to enable wire compression without a protocol
rev, like sending a gzip header before the startup packet so a server that
doesn't understand it just sees a nonsensical startup packet and borks. But
they all either boil down to uglier versions of genuine protocol bump or
are potentially subject to the issues you raised earlier with
GUC-controlled protocol changes.

In retrospect it's a huge pity there was no set of client-library-only
key/value fields defined in the startup packet in addition to GUCs that
libpq lets clients set via PGOPTIONS etc. That would've let us extend the
protocol much more easily by sending capabilities like "I understand
protocol compression".

Personally I think we should do that anyway - allow a startup-only GUC like
proto_compression=gzip .

To help protect against abuse via

PGOPTIONS="-c proto_compression=gzip" ./exploitable-old-libpq-based-binary

we could do something like sending an ErrorResponse with a newly defined
SQLSTATUS like 08P02 Client must support gzip compression . A client that
supports gzip may ignore this error and proceed normally. A client that
doesn't will not recognise the SQLSTATUS and since it's in a fatal category
08 (or rather, not the two non-fatal categories 00, 01 and 02) it'll die
and won't see any upsetting new protocol messages. Clients that don't
support it will even get a nice error message:

psql: FATAL: unrecognized configuration parameter "proto_compression"

It's really just a hack around bumping the protocol to add capability
negotiation though, and it'd scale very poorly if it was then used for more
things. Lots of reconnections needed because we can't do a two-way
negotiation.

I think retrofitting protocol compression might be compelling enough to
justify that. But it's not really significantly better than just sending a
v4 protoversion field and reconnecting if the server gets upset with that.

--
Craig Ringer http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Training & Services