Re: Key management with tests

On Fri, Feb 5, 2021 at 01:14:35PM -0500, Stephen Frost wrote:
> > I looked further. First, I don't think we are going to be able to
> > protect at all against users who have _write_ access on the OS running
> > Postgres. It would be too easy to just read process memory, or modify
> > ~/.profile.
>
> I don't think anyone is really expecting that we'll be able to come up
> with a way to protect against attackers who have fully compromised the
> OS to the point where they can read/write OS memory, or even the PG unix
> account. I'm certainly not suggesting that there is a way to do that or
> that it's an attack vector we are trying to address here.

OK, that's good.

> > I think the only possible option would be to try to give some protection
> > against users with write access to PGDATA, where PGDATA is on another
> > server, e.g., via NFS. We can't protect against all db modifications,
> > for reasons outlined above, but we might be able to protect against
> > write users being able to _read_ the keys and therefore decrypt data.
>
> That certainly seems like a worthy goal. I also really want to stress
> that I don't think anyone is expecting us to be able to "protect"
> against users who have write access to the system- write access to files
> is really an OS level issue and there's not much we can do once someone
> has found a way to circumvent that (we can try to help the OS by doing
> things like using SELinux, of course, but that's a different
> discussion). At the point that an attacker has gotten write access, the

Agreed.

> best we can do is complain loudly if we detect unexpected modifications.
> Ideally, we would be able to do that for everything, but certainly doing
> it for the principal data would go a long way and is far better than
> nothing.

I disagree. If we only warn about some parts, attackers will just
attack other parts. It will also give users a false sense of security.
If you can get the keys, it doesn't matter if there is one or ten ways
of getting them, if they are all of equal difficulty. Same with
modifying the system files.

> Now, that said, I don't know that we absolutely must have that in the
> first release of TDE support for PG. In thinking about this, I would
> say we have two basic options:

I skipped this part since I think we need a fully secure plan before
considering page format changes. We don't need it for our currently
outlined feature-set.

> > Looking at PGDATA, we have, at least:
> >
> > postgresql.conf
> > pg_hba.conf
> > postmaster.opts
> > postgresql.conf.auto
> >
> > which could be exploited to cause reading of the cluster key or process
> > memory. The first two can be located outside of PGDATA but the last two
> > currently cannot.
>
> There are certainly already users out there who intentionally make
> postgresql.auto.conf owned by root/root, zero-sized, and monitor it to
> make sure that it isn't updated. postgresql.conf actually is also often
> monitored for changes by a change management system of some kind and may
> also be owned by root/root already. I suspect that postmaster.opts is
> not monitored as closely, but that's probably due more to the fact that
> we don't really document it as a configuration system file and it can't
> be put outside of PGDATA. Having a way to move it outside of PGDATA or
> just not have it be used at all (do we really need it..?) would be
> another way to address that risk though.

I think postmaster.opts is used for pg_ctl reload. I think the question
is whether the value of maliciously writable PGDATA being able to read
the keys, while not protecting or detecting all malicious
writes/db-modifications, is worth it. And, while I listed the files
above, there are probably many more ways to break the system.

> > The problem is that this is a limited use-case, and there are probably
> > other problems I am not considering. It seems too error-prone to even
> > try protect against this, but it does limit the value of this feature.
>
> I don't think we need to consider it a failing of the capability every
> time we think of something else that really should be addressed when
> considering this attack vector. We aren't going to be releasing this
> and saying "we guarantee that this protects against an attacker who has
> write access to PGDATA". Instead, we would be documenting "XYZ, when
> enabled, is used to validate the integrity of ABC data. Individuals
> concerned with unexpected modifications to their system should consider
> independently monitoring files D, E, F. Note that there is currently no
> explicit protection against or detection of unexpected or malicious
> modification of other parts of the system such as the transaction
> record.", or something along those lines. Hardening guidelines would
> also recommend things like having postgresql.conf moved out of PGDATA
> and owned by root/root, etc. Users would then have the ability to
> evaluate if what we're providing is sufficient for their requirements
> or not, and to then provide us with feedback about what they feel is
> still missing before they would be able to use PG for their use-case.

See above --- I think we can't just say we close _most_ of the doors
here, and I am afraid there will be more and more cases we miss. It
feels too open-ended. For example, imagine modifying a PGDATA file so
it is a symbolic link to another file that is not in PGDATA? Seems that
would break all sorts of security restrictions, and that's just a new
idea I came up with today.

What I don't want to do is to add a lot of complexity to the system, and
not really gain any meaningful security.

> To that end, I would hope that we'd eventually develop a way to detect
> unexpected modifications in other parts of the system, both as a way to
> discover filesystem corruption earlier but also in the case of a
> malicious attacker. The latter would involve more work, of course, but
> it doesn't seem insurmountable. I don't think it's necessary to get
> into that today though.
>
> I am concerned when statements are made that we are just never going to
> do something-or-other because we think it'd be a lot of source code
> changes or won't be completely perfect against every attack we can think
> of. There was a good bit of that with RLS which also made it a
> particularly difficult feature to push forward, but, thanks to clearly
> documenting what was and wasn't addressed, clearly admitting that there
> are covert channel attacks that might be possible due to how it works,
> it's been pretty well accepted and there hasn't been some huge number of
> issues or CVEs that have been associated with it or mismatched
> expectations that users of it have had regarding what it does and
> doesn't protect against.

Oh, that is a very meaningful lesson. I do think that for cluster file
encryption, if we have a vulnerability, someone will write a script for
it, and it could be widely exploited. I think RLS gets a little more
flexibility since someone is already in the database when using it.

I am not against adding more security features, but I need agreement
that the existing features/protections, with the planned source code
impact, is acceptable. I don't want to go down the road of getting the
feature with the _hope_ that later changes will make the feature
acceptable --- for me, either what we are planning now is acceptable
given its code impact, or it is not. If the feature is not sufficient,
then I would not move forward until we had a reasonable plan of when the
feature would have acceptable usefulness, and acceptable source code
impact.

The big problem, as you outlined above, is that adding to the
protections, like malicious write detection for a remote PGDATA, greatly
increases the code impact, and ultimately, might be unsolvable.

--
Bruce Momjian <bruce(at)momjian(dot)us> https://momjian.us
EDB https://enterprisedb.com

The usefulness of a cup is in its emptiness, Bruce Lee