Re: Slowness of extended protocol

Shay Rojansky <roji(at)roji(dot)org>:

> That's definitely a valid point. But do you think it's a strong enough
> argument to avoid ever adding new messages?
>

The point is "adding a message to current v3 protocol is not a backward
compatible change".
The problem with adding new message types is not only "client support", but
deployment issues as well: new message would require simultaneous upgrade
of both backend, client, and pgbouncer.

It could make sense to add some kind of transparent extensibility to the
protocol, so clients can just ignore unknown message types.

For instance, consider a new message "ExtensibleMessage" is added: '#',
int16 (message id), int32 (message length), contents.
For the pgbouncer case above, pgbouncer could just proxy "unknown requests"
as is and it would cover many of the cases out of the box.
That message has a well-defined message length, so it is easy to skip.

If new "messages" required, a new message_id can be allocated and assigned
to the new message type. Old clients would be able to skip the message as
they know its length.

Of course, adding that '#' message does require support from pgbouncer,
etc, etc, however I'm sure it would simplify protocol evolution in the
future.

Of course there might appear a need for a message that cannot be ignored by
the client, but I think "even a bit of flexibility" is better than no
flexibility at all.

Technically speaking, there's a "NotificationResponse" message, however it
is not that good since it cannot have 0 bytes in the contents.

Shay Rojansky <roji(at)roji(dot)org>:

>
>> On the other hand, usage of some well-defined statement name to trigger
>> the special case would be fine: all pgbouncer versions would pass those
>> parse/bind/exec message as if it were regular messages.
>>
>
> Can you elaborate on what that means exactly? Are you proposing to somehow
> piggyback on an existing message (e.g. Parse) but use some special
> statement name that would make PostgreSQL interpret it as a different
> message type?
>

Exactly.
For instance: if client sends
Parse(statementName=I_swear_the_statement_will_be_used_only_once), then the
subsequent message must be BindMessage, and the subsequent must be
ExecMessage for exactly the same statement id.

Then backend could recognize the pattern and perform the optimization.
Note: it is quite easy to invent a name that is not yet used in the wild,
so it is safe.
From security point of view (e.g. if a client would want to exploit
use-after-free kind of issues), backend could detect deviations from this
parse-bind-exec sequence and just drop the mic off.

Shay Rojansky <roji(at)roji(dot)org>:

> Apart from being a pretty horrible hack,
>

I would not call it a horrible hack. That is just a clever use of existing
bits, and it does not break neither backward nor forward compatibility.
Backward compatibility: new clients would be compatible with old PG
versions (even 8.4).
Forward compatibility: even if the support of that special statement name
would get dropped for some reason, there will be no application issues (it
would just result in a slight performance degradation).

Shay Rojansky <roji(at)roji(dot)org>:

> it would still break pgbouncer, which has to actually inspect and
> understand SQL being sent to the database (e.g. to know when transactions
> start and stop).
>

Note: I do not suggest to change message formats. The message itself is
just fine and existing pgbouncer versions can inspect the SQL. The
difference is a special statementName, and I see no reasons for that kind
of change to break pgbouncer.

Vladimir