Re: Internal key management system

On Thu, Oct 29, 2020 at 1:22 AM Stephen Frost <sfrost(at)snowman(dot)net> wrote:

> > Most importantly - I don't think the SQL key adds anything really
> > crucial that we cannot do at the SQL level with an extension. An
> > extension "pg_wrap" could provide pg_wrap() and pg_unwrap() already,
> > using a single master key much like the SQL key proposed in this
> > patch. To store the master key it could:
>
> Lots of things can be done in extensions but, at least for my part, I'd
> much rather see us build in an SQL key capability (with things like
> grammar support and being able to tie to to a role cleanly) than to try
> and figure out how to make this work as an extension.
>

I agree with you there. I'm suggesting that this first patch focus on full
on-disk encryption, and that someone who desperately needs SQL-level keyops
could build on this patch in an extension.

I definitely don't want an extension to be the preferred / blessed way to
do those things, I'm only pointing out that deferring the SQL-level stuff
doesn't prevent someone from doing it if they need those capabilities
before a mature core patch is ready for them. Trying to roll the SQL-level
stuff into this patch will distract from getting the basics working and
either cause massive scope creep or leave us with a seriously limited
interface that will make doing it right later much harder.

+100 to having client-driver-assisted encryption, this solves real
> attack vectors which traditional TDE simply doesn't, compared to
> filesystem or block device level encryption (even though lots of
> people seem to think it does, which is bizarre to me).
>

Many things people believe about security are bizarre to me. I stopped
being surprised a long time ago...

I would think we'd want to enable admins to be able to control if what
> is being provided is a KEK (where the key is then decrypted by PG and PG
> then uses whatever libraries it's built with to perform the encryption
> and decryption in PG process space), or an engine/offloading
> configuration (where PG doesn't ever see the actual key and all
> encryption and decryption is done outside of PG's control by an HSM or
> the Linux kernel through the crypto API or whatever).
>

I had that in mind too, but deliberately did not raise it because I don't
think it's necessary to address that when introducing the basics of full
on-disk encryption.

I just don't think there are enough users who both have access to a high
performance PCIe or SoC based crypto offload engine and could tolerate the
limited database throughput they'd get when using even the most optimised
crypto offload engine out there. Most HSMs are optimised for SSL/TLS and
for asymmetric crypto ops using RSA etc, plus small-packet AES. There are
also crypto offload cards for high throughput bulk symmetric AES etc but
they don't all have HSM-like secrecy features, plus the cost tends to be
absolutely staggering.

So I thought it made sense to focus on the KEK for now. I don't think
managing the WAL and heap keys in a HSM is a realistic use case for all but
the tinest possible set of users, and the complexity we'd have to deal with
in terms of key rotations etc would be much greater.

The use-cases I'm thinking about:
>
> - User has a Yubikey, but would like PG to be able to write more than
> one block at a time. In this case, the Yubikey would have a KEK which
> PG doesn't ever see.

Yes. This is the main case I'm focused on making it possible to add support
for. Not necessarily in the first cut of this patch, but I want to at least
ensure that this patch doesn't bake in so many assumptions about the KEK
that it'd be really hard to add external KEK management later.

PG would have an encrypted blob that it then
> asks the yubikey to decrypt which contains the actual key that's then
> kept in PG's memory to perform the encryption/decryption. Naturally,
> if that key is stolen then an attacker could decrypt the entire
> database, even if they don't have the yubikey. An attacker could
> acquire that key by having sufficient access on the PG sever to be
> able to read PG's memory.
>

Correct. Or they could gain the rights to run code as the postgres unix
user and ask the HSM to decrypt the cluster keys for them - assuming the
HSM doesn't have any external authorization channel or checks, like PIN
entry, touch-test for physical access, or the like.

For that to actually be useful they also have to have a copy of the
database's on-disk representation - copy it off, steal a backup, etc. If
they gained enough access to copy the whole DB off they can probably just
as easily pg_dump it though; the only way to prevent that kind of attack is
to use client-driver-side encryption which is a totally different topic.

Stealing a backup then separately breaking into a running instance with
matching keys to steal the key is a pretty high bar to set.

The main weakness here is with replicas. But it doesn't really matter if
the replicas have the same heap and WAL keys as the primary or not, if the
attacker compromises one replica your data is still exposed.

- User has a Thales Luna PCIe HSM, or similar. In this case, the user
> wants *all* of the encryption/decryption happening on the HSM and none
> of it happening in PG space, making it impossible for an attacker to
> acquire the actual key.
>

Right. They can still pg_dump it, or trick Pg into decrypting it for them
in other ways, but they cannot steal the key then use it to decrypt a
stolen copy of the DB itself.

Per above, though, I don't think this actually adds all that much real
world security over protecting the KEK.

I mean - face it, building database encryption into postgres itself isn't
that much stronger than doing it at the filesystem level in a LUKS volume
or similar. It's a marketing thing as much as a real world security
benefit. The interesting parts only really come when the DB doesn't even
have access to the keys (client-driver encryption) or only has transient
access to the keys (session-level client secret key unlock), neither of
which are within the scope of this proposed patch.

Handling encryption at the Pg level is mainly nice for backup protection.

To be clear I'm not against making this possible, and think it should
actually be relatively simple to do if we use proper key ops abstractions,
I just don't think it's all that interesting or important. It could also
get very hairy when dealing with postgres's fork() based processing...

- User has a yubikey, similar to #1, but would like to have the Linux
> kernel used to safe-guard the actual key used.
>

That really works the same as #2, Pg is using some kind of engine to handle
crypto ops on the WAL and heap keys rather than doing them in-process
itself. It doesn't matter what the engine is or where it lives - in
software in the kernel, in a PCIe card that costs more than a luxury car,
or whatever else.

- User has a vaulting solution, and perhaps wants to store the actual
> encryption/decryption key there, or perhaps the user wants to store a
> passphrase in the vault and have PG derive the actual key from that.
> Either seems like it could be reasonable.
>

Sure. Storing the KEK-generation passphrase in a vault is possible with the
proposed approach as-is.

If they want to store the actual KEK in a vault they could do so in much
the same way as a HSM or anything else, and Pg does not have to care. So
long as we provide a way to plug in KEK loading and we only do KEK
crypt/decrypt/verify ops via an API like the one we have already for
PgCipherCtx it just doesn't matter exactly where the KEK lives.

Instead of

cluster_crypto_method = 'openssl_engine'

to load cluster_crypto_openssl_engine.so and have that provide the
PgKeyWrapCtx with PgCipherCtx, you'd

cluster_crypto_method = 'loadkey'

and have the cluster_crypto_loadkey.so accept a keyfile path or read-key
command.

Personally I think the default method of generating a kek from a passphrase
should be behind the same kind of abstraction, but I don't get to have a
strong opinion on that if I am not currently prepared to write all the code
for it.

What I'm wondering about here is if we should make it an explicit option
> for a user to pick through the server configuration about if they're
> giving PG a direct key to use, a KEK that's actually meant to decrypt
> the data key, a way to fetch the direct key or the KEK, or a engine
> which has the KEK to ask to decrypt the data key, etc.

-1

We can't anticipate all the things users will want, and if we try we'll
land up with a horribly complex set of configuration options.

We should provide an interface people can use to implement and load what
they want to do, then provide a simple default implementation that does the
basic passphrase based setup.

Want anything else? Load a plugin.

That way we aren't stuck supporting some weird and random openssl-specific
GUCs once we eventually support host crypto libraries. And re-keying the
KEK becomes as simple as "load new KEK module and write the cluster keys
using the new KEK module". Code for re-keying etc doesn't have to know all
the details.

This approach was taken for logical decoding and I think it was 100% the
right one. We should go for something like it here too.

I don't want to go full plugin crazy. I've used Jenkins, I know the pain
that "everything is a plugin" brings. But in the right places, and with
good default plugin implementations bundled with the server (like we have
with pgoutput) having plugin interfaces at the correct boundaries works
really well.

If we can come
> up with a way to configure PG that will support the different use cases
> outlined above without being overly complicated, that'd be great. I'm
> not sure that I see that in what you've proposed here, but maybe by
> going through each of the use-cases and showing how a user would
> configure PG for each with this proposal, I will.
>

Interactive password prompt, using Bruce's %R file descriptor passing:

cluster_encryption = 'password'
cluster_encryption_password.password_command = ' IFS=$'\n' read -s -p
"Prompt: " -r -u %R PASS && echo $PASS '

which will do the same as this:

$ IFS=$'\n' read -s -p "Prompt: " -r -u 0 PASS ; echo; echo $PASS
Prompt:
pass word here

A pretty script or default command would obviously be appropriate here, I'm
just showing how basic it is.

The same thing as above would work for a vault tool that pases the key on
stdin, or that passes a file descriptor for an unlinked tempfile the
password can be read from.

Password fetched by obfuscated command or from some vault tool etc:

cluster_encryption = 'password'
cluster_encryption_password.password_command =
'/usr/bin/read-my-secret-password'

Read key from a file on a short-lived mount, usb key that's physically
removed after loading, or whatever:

cluster_encryption = 'keyfile'
cluster_encryption_keyfile.key_file = '/mnt/secretusb/key.pem'

Read whole key from a command, vault tool, etc in case you wanted that
instead:

cluster_encryption = 'keyfile'
cluster_encryption_keyfile.key_command = '/bin/my-vault-tool get-key
foo'

Use AWS CloudHSM for your KEK:

cluster_encryption = 'openssl_engine'
cluster_encryption_openssl.engine = 'cloudhsm'
cluster_encryption_openssl.key = 'mycloudkeyname'

Keep the key in the host TPM and use it to perform KEK ops, assuming you
have p11-kit and you generated a key in the TPM with the tpm2 tools:

cluster_encryption = 'openssl_engine'
cluster_encryption_openssl_engine.engine = 'pkcs11'
cluster_encryption_openssl_engine.key =
'pkcs11:module-path=/usr/lib64/pkcs11/libtpm2_pkcs11.so;model=TPM2'

Keep the key in an OpenSC-supported smartcard or key like a yubikey and use
it via OpenSC to perform KEK ops, once the key is appropriately configured
with the card tools and assuming p11-kit:

cluster_encryption = 'openssl_engine'
cluster_encryption_openssl_engine.engine = 'pkcs11'
cluster_encryption_openssl_engine.key =
'pkcs11;module-path=/usr/lib64/pkcs11/opensc-pkcs11.so;token=%2FCN%3Dpg%2F'

... etc

I agree that it doesn't seem like a bad approach to expose that URI, but
> I'm not sure that's really the end of it since there's going to be cases
> where people would like to have a KEK on a yubikey and there'll be other
> cases where people would like to offload all of the encryption and
> decryption to a HSM crypto accelerator and, ideally, we'd allow them to
> be able to configure PG for either of those cases.
>

Sure, eventually.

I don't think it's necessarily that hard either. If you wanted you could
probably put the WAL and heap key acquisition behind a pluggable interface
too, and use the same KeyWrapCtx and PgCipherCtx to abstract their use.

fork() could be exciting, but mostly that's a matter of adding before-fork
and after-fork APIs to let the plugin do the right thing depending on the
underlying library it uses.

I don't see a problem with adding hooks, where they make sense, but we
> should also make things work in a sensible way and a way that works with
> at least the use-cases that I've outlined, ideally, without having to go
> get an extension or write C code.
>

I think the sensible use case *is* the generated password, simple
configuration.

What I'd ideally like to do is have that as a sort of default
cluster_encryption_plugin called 'password' per the imaginary config I
outlined above.

Then we could bundle an openssl_engine plugin that would let you do pretty
much anything else by configuring openssl, using openssl engines directly
or via pkcs#11, etc.

> There's an active patch that's been worked on for quite some time that's
> getting some renewed interest in adding NSS support, something I
> certainly support also, so we really shouldn't be taking steps that end
> up making it more difficult to support alternatives.

Right.

So in the plugin based approach above that would mean providing a
cluster_encryption_plugin='nss' .

If we extend PgCipherCtx to support HMAC it should be fairly
straightforward.

I definitely think we want to support things directly in PG and not
> require an extension or something to be in s_p_l for this.
>

Alternative proposed above - support dynamic loading but use a separate
entrypoint. And if we want we can compile in "plugins" anyway. The
interface should be the same whether dynamically loaded or baked in.

> But you might not even have the key. In some HSM implementations the
> > key is completely sealed - you can program new HSMs to have the same
> > key by using the same configuration, but you cannot actually obtain
> > the key short of attacks on the HSM hardware itself. That's very much
> > by design - the HSM configuration is usually on an air-gapped system,
> > and it isn't sufficient to decrypt anything unless you also have
> > access to a copy of the HSM hardware itself. Obviously you accept the
> > risks if you take that approach, and you must have an escape route
> > where you can re-encrypt the material protected by the HSM against
> > some other key. But it's not at all uncommon.
>
> Right, but in such cases you'd need an HSM that's able to perform
> encryption and decryption at some reasonable rate.
>

No, you just have to use it to decrypt and load the WAL and heap keys at
startup.

I understand why you're exploring the idea of full crypto offload, but I
personally think it's premature. However the same sorts of things that
would allow HSM use instead of a password would also be necessary steps
toward what you propose.