Re: md5 issues Postgres14 on OL7

Michael Paquier <michael(at)paquier(dot)xyz> writes:
> On Mon, Dec 20, 2021 at 03:22:31PM +0100, Christoph Moench-Tegeder wrote:
>> Active FIPS mode (/proc/sys/crypto/fips_enabled => 1) on the server does
>> produce this behaviour.

> Most likely, this is a build linked with OpenSSL? The way MD5 hashes
> are computed in Postgres has largely changed in 14, and the code has
> been refactored so as we rely on the EVP APIs from OpenSSL when
> building with --with-ssl=openssl, having as direct consequence to
> allocate a bit more memory every time a hash is computed. My guess is
> that this comes from pg_cryptohash_create() in cryptohash_openssl.c,
> with a complain coming from OpenSSL's EVP_MD_CTX_create(), but there
> are other palloc() calls in this area as well.

I reproduced this on Fedora 35 with FIPS mode enabled. The problem
is that OpenSSL treats MD5 as a disallowed cipher type under FIPS
mode, so this call in pg_cryptohash_init fails:

status = EVP_DigestInit_ex(ctx->evpctx, EVP_md5(), NULL);

and then we come back to this in md5_text():

/* get the hash result */
if (pg_md5_hash(VARDATA_ANY(in_text), len, hexsum) == false)
ereport(ERROR,
(errcode(ERRCODE_OUT_OF_MEMORY),
errmsg("out of memory")));

So there's nothing actually misbehaving, but our error reportage sucks:
the hash functions have no way to report a specific failure code,
and the caller(s) think the only possible failure mode is OOM.

I suppose we could get around the error by using our own MD5 code
even in OpenSSL-enabled builds, but that'd violate both the spirit
and the letter of FIPS certification. I think the right response is
to upgrade the error-reporting API in this area, so that the message
could look more like "MD5 is disallowed in FIPS mode".

regards, tom lane