Proposal: http2 wire format

Hello hackers,

I’d like to propose the implementation of new wire protocol using http2 framing.

It appears to me that http2 solves many of the issues on the TODO list under “Wire Protocol Changes / v4 Protocol,“ without any obvious downsides.

The implementation I have in mind has zero impact on existing clients. No changes to the format of existing v3 protocol. The new protocol works through a few small additions to postmaster.c to intercept TLS requests, and the rest in new source files, linked through PQcommMethods.

I’d like to emphasize that this proposal is empathically NOT about “let’s handle REST in the database” or some such. It’s about upgrading the framing, where http2 offers many benefits: content negotiation, concurrent bidirectional streams, extensible frame types, metadata/data split into headers/trailers and data frames, flow control, etc. It’s at least as efficient as febe v3. A lot of research is going into it to make it even more efficient and latency friendly. The mechanisms it provides for content negotiation, (and with ALPN, protocol negotiation), offers us a future-friendly way to evolve without the burden of backward compatibility compromises.

Before writing this proposal, I set out to create a proof of concept. My goal for the PoC is to be able to connect to the server using an existing http2 client and get json back:

curl -k https://localhost:5432/some_func \
--http2-prior-knowledge --tlsv1.2 \
-H 'pg-database: postgres' \
-H 'pg-user: web' \
-H ‘authorization: ….’
-H ‘accept: application/json’

{ result: [ … ] }

After spending a week getting up to speed with C, libpq internals, http2 standard, libnghttp2 interface, etc., I’m fairly convinced that pg/http2 is feasible.

Sadly, my experience with C and Postgres internals is non-existent, and I am not yet able to finalize a live demo. The above curl request does establish the connection, receives the settings frame and queries the database, but I’m still struggling with writing code to return the http2 response. At this stage, it’s purely an issue of mechanically writing the code, I think I solved how it all works in principle.

If anyone finds the idea of Postgres speaking http2 appealing, I’d welcome guidance/mentoring/coding help (or just plain taking over). I a put up a repo with the results so far and a longer writeup: https://github.com/dsimunic/pg_h2

All changes I made to the codebase are in a single commit, hopefully easy to understand what is happening. You’ll need libnghttp2 and openssl 1.0.2 or newer to compile.

My hope is that this post leads to a conversation and gets a few people excited about the idea the way I am. Maybe even some of the GSoC students would take the implementation further?

Damir