Support for Secure Transport SSL library on macOS as OpenSSL alternative

In https://postgr.es/m/69DB7657-3F9D-4D30-8A4B-E06034251F61@yesql.se I
presented a WIP patch for adding support for the Apple Secure Transport SSL
library on macOS as, an alternative to OpenSSL. That patch got put on the
backburner for a bit, but I’ve now found the time to make enough progress to
warrant a new submission for discussions on this (and hopefully help hacking).

It is a drop-in replacement for the OpenSSL code, and supports all the same
features and options, except for two things: compression is not supported and
the CRL cannot be loaded from a plain PEM file. A Keychain must be used for
that instead.

Current state
=============
The frontend and backend are implemented more or less fully, the two main
things missing being the CRL support (further details below) and loading DH
files (to support the GUC added in c0a15e07cd). All non-CRL tests but one are
passing. The failing test is in the frontend and it also fails when running
against an OpenSSL backend, but I can’t find where the logic is flawed and
could do with some help there.

Threads
=======
Just like the CFLocaleCopyCurrent() call referenced in postmaster.c, the Secure
Transport APIs makes the process multithreaded. To keep this out of the
postmaster, and contained in the backend, I’ve moved all functionality to
open_server and left init empty. I could definitely need some clues on how to
properly handle this, or if it’s a complete showstopper.

Keychains
=========
The frontend has support for using PEM files for certificates and keys. It can
also look up the key for the certificate in a Keychain, or both certificate and
key in a Keychain. The root certificate is still just read from a PEM file.
The existence of an sslcert config trumps a keychain, but when a keychain is
used I’m currently using the sslcert parameter (awaiting a discussion on how to
properly do this) for the certificate label required to search the keychain.

There is a new frontend parameter called “keychain” with which a path to a
custom Keychain file can be passed. If set, this Keychain will be searched as
well as the default. If not, only the default user Keychain is used. There is
nothing that modifies the Keychain in this patch, it can read identities
(certificates and its key) but not alter them in any way.

The backend is only supporting PEM files at this point.

Once we have support for Keychains, we can however use them for additionally
supporting other Secure Transport features like OCSP etc.

“keychain” is obviously a very Secure Transport specific name, and I personally
think we should avoid that. Any new configuration added here should take
future potential implementation into consideration such that avoid the need for
lots of backend specific knobs. “sslcertstore” comes to mind as an
alternative, but we’d also need parameters to point into the certstore for
finding what we need. Another option would be to do a URL based scheme
perhaps.

Certificate Revocation
======================
Secure Transport supports loading CRL lists into Keychain files, the command
line certtool can for example do that. When doing the trust evaluation on the
connection, a policy can be added which enables revocation checking via CRL. I
have however been unable to figure out how to load a CRL list programmatically,
and as far as I can tell there is no API for this. The certtool utility is
using the low-level CSSM APIs for this it seems, but adding that level of
complexity doesn’t seem worth it for us to maintain (I did a test and it turned
big, ugly and messy).

Unless someone knows how to implement this, we’d be limited to requiring the
use of a Keychain file for CRL support. This would break drop-in replacement
support, but supporting certificate revocation seems more important.

Platform Support
================
I’ve tested this on 10.11 El Capitan and 10.12 Sierra, which are the systems I
have access to. Supporting prairiedog and dromedary in the buildfarm will
probably be too hard (if at all possible), but down to 10.9 Mavericks should be
quite possible (if someone has the required systems to test on). It adds a
dependency on Core Foundation on top of Secure Transport, no other macOS APIs
are used.

Testing
=======
In order to test this we need to provide an alternative to the openssl calls in
src/test/ssl/Makefile for Secure Transport. On top of that, code to generate
Keychains is required. The certtool application can do all the Keychain
generations (I think) but this is still left open. The main thing to figure
out here is how to avoid having to type in the Keychain password in a modal GUI
that Secure Transport pops up. Since a Keychain can be passwordless it should
be doable, but the right certtool incantations for that is still evading me.
I’ve included a show-and-tell patch for this which I’ve used locally for
testing during hacking.

Documentation
=============
I have started fiddling with this a little, but to avoid spending time on the
wrong thing I have done very little awaiting the outcome of discussions here.
I have tried to add lots of comments to the code however, to explain the quirks
of Secure Transport.

I went into this thinking I would write a README for how to implement a new SSL
library. But in the end, turns out the refactoring that went into our SSL code
earlier made that part almost too easy to warrant that. It’s really quite
straightforward.

Patches
=======
0001 - Adds support for running the SSL tests against another set of server
binaries. This is only useful for testing during the implementation of a new
SSL library, but then it’s crucial. Nothing Secure Transport specific in this
patch.

0002 - Secure Transport fronten and backend as well as required scaffolding for
building and a new GUC to show the current backend.

securetransport_test.diff - Some ugly hacks I’ve used for testing, included as
a show-and-tell, not for any form of submission.

This leaves quite a few open questions that I hope to get some feedback on, and
some code issues which I’d love eyes/hacking on. Do we still want this, and if
so how to handle that we can’t be fully drop-in compatible with OpenSSL with
regards to using a PEM file only configuration? Should we support PEM files at
all in the backend or only Keychains? Another option could be to support
PKCS12 files instead of (or additionally to) PEM since there is likely to be
API support for loading PKCS12 and they can afaik contain CRLs. How can we
ensure that new parameters hopefully covers more libraries than just Secure
Transport?

cheers ./daniel