Re: [Proposal] Table-level Transparent Data Encryption (TDE) and Key Management Service (KMS)

Bruce Momjian <bruce(at)momjian(dot)us> wrote:

> On Sat, Aug 10, 2019 at 08:06:17AM -0400, Bruce Momjian wrote:
> > So, I just had an indea if we use separate encryption keys for
> > heap/index and for WAL --- we already know we will have an offline tool
> > that can rotate the passphrase or encryption keys. If we allow the
> > encryption keys to be rotated independently, we can create a standby,
> > and immediately rotate its heap/index encryption key. We can then start
> > streaming replication. When we promote the standby to primary, we can
> > then shut it down and rotate the WAL encryption key --- the new primary
> > would then have no shared keys with the old primary.
>
> To help move this forward, I created a new wiki TDE section titled "TODO
> for Full-Cluster Encryption" and marked some unresolved items with
> question marks:
>
> https://wiki.postgresql.org/wiki/Transparent_Data_Encryption#TODO_for_Full-Cluster_Encryption
>
> I have also updated some of the other text to match conclusions we have
> made.
>
> I know some of the items are done, but if we have agreement on moving
> forward, I can help with some of the missing code. This looks doable
> for PG 13 if we start soon.

I can work on it right away but don't know where to start.

First, I think we should use a code repository to integrate [1] and [2]
instead of sending diffs back and forth. That would force us to resolve
conflicts soon and help to avoid duplicate work. The diffs would be created
only whe we need to post the next patch version to pgsql-hackers for review,
otherwise the discussions of details can take place elsewhere.

A separate branch can be created for the Full-Cluster Encryption at some point
- there are probably multiple branching strategies.

The most difficult problem I see now regarding the collaboration is agreement
on the key management user interface. The Full-Cluster Encryption feature [1]
should not add configuration variables or even tools that the next, more
sophisticated version [2] deprecates immediately. Part of the problem is that
[2] puts all (key management related) interaction of postgres with the
environment into an external library. As I pointed out in my response to [2],
this will not work for frontend applications (e.g. pg_waldump). I think the
key management UI for [2] needs to be designed first even if PG 13 should
adopt only [1].

At least it should be clear how [2] will retrieve the master key because [1]
should not do it in a differnt way. (The GUC cluster_passphrase_command
mentioned in [3] seems viable, although I think [1] uses approach which is
more convenient if the passphrase should be read from console.) Rotation of
the master key is another thing that both versions of the feature should do in
the same way. And of course, the fronend applications need consistent approach
too.

I'm not too happy to start another (potentially long) discussion in this
already huge thread, but I think the UI stuff belongs to the -hackers list
rather than to an offline discussion.

[1] https://commitfest.postgresql.org/23/2104/

[2] https://www.postgresql.org/message-id/CAD21AoBjrbxvaMpTApX1cEsO%3D8N%3Dnc2xVZPB0d9e-VjJ%3DYaRnw%40mail.gmail.com

[3]
https://wiki.postgresql.org/wiki/Transparent_Data_Encryption#TODO_for_Full-Cluster_Encryption

--
Antonin Houska
Web: https://www.cybertec-postgresql.com