Password identifiers, protocol aging and SCRAM protocol

Hi all

As a continuation of the thread firstly dedicated to SCRAM:
http://www.postgresql.org/message-id/55192AFE.6080106@iki.fi
Here is a new thread aimed at gathering all the ideas of this previous
thread and aimed at clarifying a bit what has been discussed until now
regarding password protocols, verifiers, and SCRAM itself.

Attached is a set of patches implementing a couple of things that have
been discussed, so let's roll in. There are a couple of concepts that
are introduced in this set of patches, and those patches are aimed at
resolving the following things:
- Introduce in Postgres an extensible password aging facility, by
having a new concept of 1 user/multiple password verifier, one
password verifier per protocol.
- Give to system administrators tools to decide unsupported protocols,
and have pg_upgrade use that
- Introduce new password protocols for Postgres, aimed at replacing
existing, say limited ones.
Note that here is not discussed the point of password verifier
rolling, which is the possibility to have multiple verifiers of the
same protocol for the same user (this maps with the fact that
valid_until is still part of pg_authid here, but in order to support
authentication rolling it would be necessary to move it to
pg_auth_verifiers).

Here is a short description of each patch and what they do:
1) 0001, removing the password column from pg_authid and putting it
into a new catalog called pg_auth_verifiers that has the following
format:
- Role OID
- Password protocol
- Password verifier
The protocols proposed in this patch are "plain" and "md5", which map
to the current things that Postgres has, so there is nothing new. What
is new is the new clause PASSWORD VERIFIERS usable by CREATE/ALTER
USER, like that:
ALTER ROLE foo PASSWORD VERIFIERS (md5 = 'foo', plain = 'foo');
This is easily extensible as new protocols can be added on top of
that. This has been discussed in the previous thread.
As discussed as well previously, password_encryption is switched from
a boolean switch to a list of protocols, which is md5 by default in
this patch.
Also, as discussed in 6174(dot)1455501497(at)sss(dot)pgh(dot)pa(dot)us, pg_shadow has
been changed so as the password value is replaced by '*****'.
This patch adds docs, regression tests, pg_dump support, etc.

2) 0002, introduction of a new GUC parameter password_protocols
(superuser-only) aimed at controlling the password verifiers of
protocols that can be created. This is quite simple: all the protocols
specified in this list define what are the protocols allowed when
creating password verifiers using CREATE/ALTER ROLE. By default, and
in this patch, this is set to 'plain,md5', which is the current
default in Postgres, though a system admin could set it to 'md5', to
forbid the creation of unencrypted passwords for example. Docs and
regressions are added on the stack, the regression tests taking
advantage of the fact that this is a superuser parameters.
This patch is an answer to remarks done in the last thread regarding
the fact that there is no way to handle how a system controls what are
the password verifier types created, and protocol aging gets its sense
with with patch and 0003...

3) 0003, Introduction of a system function, that I called
pg_auth_verifiers_sanitize, which is superuser-only, aimed at cleaning
up password verifiers in pg_auth_verifiers depending on what the user
has defined in password_protocols. This basically does a heap scan of
pg_auth_verifiers, and deletes the tuple entries that are of protocols
not listed in password_protocols. I have hesitated to put that in
pg_upgrade_support.c, perhaps it would make more sense to have it
there, but feedback is welcome. I have in mind that it is actually
useful for users to have this function at hand to do post-upgrade
cleanup operations. Regression tests cannot be added for this one, I
guess the reason to not have them is obvious when considering
installcheck...

4) 0004, Have pg_upgrade make use of the system function introduced by
0003. This is quite simple, and this allows pg_upgrade to remove
entries of outdated protocols.

Those 4 patches are aimed at putting in-core basics for the concept I
call password protocol aging, which is a way to allow multiple
password protocols to be defined in Postgres, and aimed at easing
administration as well as retirement of outdated protocols, which is
something that is not doable now in Postgres.

The second set of patch 0005~0008 introduces a new protocol, SCRAM.
This is a brushed up, rebased version of the previous patches, and is
divided as follows:
5) 0005, Move of SHA1 routines of pgcrypto to src/common to allow
frontend authentication code path to use SHA1.
6) 0006 is a refactoring of sendAuthRequest that taken independently
makes sense.
7) 0007 is a small refactoring of RandomSalt(), to allow this function
to handle salt values of different lengths
8) 0008 is another refactoring, moving a set of encoding routines from
the backend's encode.c to src/common, escape, base64 and hex are moved
as such, though SCRAM uses only base64. For consistency moving all the
set made more sense to me.
9) 0009 is the SCRAM authentication itself....

The first 4 patches obviously are the core portion that I would like
to discuss about in this CF, as they put in the base for the rest, and
will surely help Postgres long-term. 0005~0008 are just refactoring
patches, so they are quite simple. 0009 though is quite difficult, and
needs careful review because it manipulates areas of the code where it
is not necessary to be an authenticated user, so if there are bugs in
it it would be possible for example to crash down Postgres just by
sending authentication requests.
Regards,
--
Michael