Re: Extensibility of the PostgreSQL wire protocol

> On 11 Feb 2021, at 16:06, Tom Lane <tgl(at)sss(dot)pgh(dot)pa(dot)us> wrote:
>
> Maybe there is some useful thing that can be accomplished here, but we
> need to consider the bigger picture rather than believing (without proof)
> that a few hook variables will be enough to do anything.
>
> regards, tom lane
>

Pluggable wire protocol is a game-changer on its own.

The bigger picture is that a right protocol choice enables large-scale architectural simplifications for whole classes of production applications.

For browser-based applications (lob, saas, e-commerce), having the database server speak the browser protocol enables architectures without backend application code. This in turn leads to significant reductions of latency, complexity, and application development time. And it’s not just lack of backend code: one also profits from all the existing infrastructure like per-query compression/format choice, browser connection management, sse, multiple streams, prioritization, caching/cdns, etc.

Don’t know if you’d consider it as a proof, yet I am seeing 2x to 4x latency reduction in production applications from protocol conversion to http/2. My present solution is a simple connection pooler I built on top of Nginx transforming the tcp stream as it passes through.

In a recent case, letting the browser talk directly to the database allowed me to get rid of a ~100k-sloc .net backend and all the complexity and infrastructure that goes with coding/testing/deploying/maintaining it, while keeping all the positives: per-query compression/data conversion, querying multiple databases over a single connection, session cookies, etc. Deployment is trivial compared to what was before. Latency is down 2x-4x across the board.

Having some production experience with this approach, I can see how http/2-speaking Postgres would further reduce latency, processing cost, and time-to-interaction for applications.

A similar case can be made for IoT where one would want to plug an iot-optimized protocol. Again, most of the benefit is possible with a protocol-converting proxy, but there are additional non-trivial performance gains to be had if the database server speaks the right protocol.

While not the only use cases, I’d venture a guess these represent a sizable chunk of what Postgres is used for today, and will be used even more for, so the positive impact of a pluggable protocol would be significant.

--
Damir