Serverside SNI support in libpq

SNI was brought up the discussions around the ALPN work, and I have had asks
for it off-list, so I decided to dust off an old patch I started around the
time we got client-side SNI support but never finished (until now). Since
there is discussion and thinking around how we handle SSL right now I wanted to
share this early even though it will be parked in the July CF for now. There
are a few usecases for serverside SNI, allowing for completely disjoint CAs for
different hostnames is one that has come up. Using strict SNI mode (elaborated
on below) as a cross-host attack mitigation was mentioned in [0].

The attached patch adds serverside SNI support to libpq, it is still a bit
rough around the edges but I'm sharing it early to make sure I'm not designing
it in a direction that the community doesn't like. A new config file
$datadir/pg_hosts.conf is used for configuring which certicate and key should
be used for which hostname. The file is parsed in the same way as pg_ident
et.al so it allows for the usual include type statements we support. A new
GUC, ssl_snimode, is added which controls how the hostname TLS extension is
handled. The possible values are off, default and strict:

- off: pg_hosts.conf is not parsed and the hostname TLS extension is
not inspected at all. The normal SSL GUCs for certificates and keys
are used.
- default: pg_hosts.conf is loaded as well as the normal GUCs. If no
match for the TLS extension hostname is found in pg_hosts the cert
and key from the postgresql.conf GUCs is used as the default (used
as a wildcard host).
- strict: only pg_hosts.conf is loaded and the TLS extension hostname
MUST be passed and MUST have a match in the configuration, else the
connection is refused.

As of now the patch use default as the initial value for the GUC.

The way multiple certificates are handled is that libpq creates one SSL_CTX for
each at startup, and switch to the appropriate one when the connection is
inspected. Configuration handling is done in secure-common to not tie it to a
specific TLS backend (should we ever support more), but the validation of the
config values is left for the TLS backend.

There are a few known open items with this patch:

* There are two OpenSSL callbacks which can be used to inspect the hostname TLS
extension: SSL_CTX_set_tlsext_servername_callback and
SSL_CTX_set_client_hello_cb. The documentation for the latter says you
shouldn't use the former, and the docs for the former says you need it even if
you use the latter. For now I'm using SSL_CTX_set_tlsext_servername_callback
mainly because the OpenSSL tools themselves use that for SNI.

* The documentation is not polished at all and will require a more work to make
it passable I think. There are also lot's more testing that can be done, so
far it's pretty basic.

* I've so far only tested with OpenSSL and haven't yet verified how LibreSSL
handles this.

--
Daniel Gustafsson

[0] https://www.postgresql.org/message-id/e782e9f4-a0cd-49f5-800b-5e32a1b29183%40eisentraut.org