Extensibility of the PostgreSQL wire protocol

The following is a request for discussion and comments, not a refined
proposal accompanied by a working patch.

As recently publicly announced Amazon Web Services is working on Babelfish,
a set of extensions that will allow PostgreSQL to be compatible with other
database systems. One part of this will be an extension that allows
PostgreSQL to listen on a secondary port and process a different wire
protocol. The first extension we are creating in this direction is handling
of the Tabular Data Stream (TDS), used by Sybase and Microsoft SQL-Server
databases. It is more efficient to build an extension, that can handle the
TDS protocol inside the backend, than creating a proxy process that
translates from TDS to libpq protocol and back.

Creating the necessary infrastructure in the postmaster and backend will
open up more possibilities, that are not tied to our compatibility efforts.
Possible use cases for wire protocol extensibility include the development
of a completely new, not backwards compatible PostgreSQL protocol or
extending the existing wire protocol with things like 3rd party connection
pool specific features (like transfer of file descriptors between pool and
working backend for example).

Our current plan is to create a new set of API calls and hooks that allow
to register additional wire protocols. The existing backend libpq
implementation will be modified to register itself using the new API. This
will serve as a proof of concept as well as ensure that the API definition
is not slanted towards a specific protocol. It is also similar to the way
table access methods and compression methods are added.

A wire protocol extension will be a standard PostgreSQL dynamic loadable
extension module. The wire protocol extensions to load will be listed in
the shared_preload_libraries GUC. The extension's Init function will
register a hook function to be called where the postmaster is currently
creating the libpq server sockets. This hook callback will then create the
server sockets and register them for monitoring via select(2) in the
postmaster main loop, using a new API function. Part of the registration
information are callback functions to invoke for accepting and
authenticating incoming connections, error reporting as well as a function
that will implement the TCOP loop for the protocol. Ongoing work on the TDS
protocol has shown us that different protocols make it desirable to have
separate implementations of the TCOP loop. The TCOP function will return
only after the connection has been terminated. Fortunately half the
interface already exists since the sending of result sets is implemented
via callback functions that are registered as the dest receiver, which
works pretty well in our current code.

Regards, Jan

--
Jan Wieck
Principal Database Engineer
Amazon Web Services