postgresql 9.3.10, FIPS mode and DRBG issues.

Sorry for this repost: I forgot the subject line! My bad ... :-<

Hi, there.

First, my particulars:

* Ubuntu Trusty build and runtime environment

* PostgreSQL 9.3.10 Ubuntu source code

* Using a FIPS enabled version of OpenSSL (i.e. 1.0.1p version of the library and 2.0.9 of the FIPS canister source code)

* I initially posted this to the pgsql-general list a few weeks ago, but I wasn't able to get enough specific information to resolve my issues. Hence, why I am posting this here.

* I am new to FIPS and postgresql in general (i.e. working with them for a few months)

I've been trying to get the postgresql packages to work in FIPS mode. To accomplish this, I've patched the Ubuntu source code with the patch that is attached to this message.

The main postgresql server runs fine as expected in either FIPS or non-FIPS modes. However, when I try to use the psql command in FIPS mode, I get the following error:

# psql -h 127.0.0.1 -U postgres -d sslmode=require
psql: SSL SYSCALL error: EOF detected

I used the gdb debugger to try to find where in the backend the command was failing. The backtrace on the server side suggests that the problem involves the key-exchange failing:

(gdb) bt
#0 0x00007f40183e8f20 in __nanosleep_nocancel () at ../sysdeps/unix/syscall-template.S:81
#1 0x00007f40183e8dd4 in __sleep (seconds=0) at ../sysdeps/unix/sysv/linux/sleep.c:137
#2 0x00007f40196a95ce in DH_generate_key () from /usr/lib/x86_64-linux-gnu/libcrypto.so.1.0.0
#3 0x00007f40199e8ba6 in ssl3_send_server_key_exchange () from /usr/lib/x86_64-linux-gnu/libssl.so.1.0.0
#4 0x00007f40199ec18b in ssl3_accept () from /usr/lib/x86_64-linux-gnu/libssl.so.1.0.0
#5 0x00007f40199fb8b3 in ssl23_accept () from /usr/lib/x86_64-linux-gnu/libssl.so.1.0.0
#6 0x00005618082567a4 in open_server_SSL (port=0x561808e05700) at /home/rlott/git/stash/postgresql-fips/postgresql-9.3-9.3.10/build/../src/backend/libpq/be-secure.c:925
#7 secure_open_server (port=port(at)entry=0x561808e05700) at /home/rlott/git/stash/postgresql-fips/postgresql-9.3-9.3.10/build/../src/backend/libpq/be-secure.c:221
#8 0x00005618082c7eb8 in ProcessStartupPacket (port=port(at)entry=0x561808e05700, SSLdone=SSLdone(at)entry=0 '\000') at /home/rlott/git/stash/postgresql-fips/postgresql-9.3-9.3.10/build/../src/backend/postmaster/postmaster.c:1921
#9 0x00005618081030f9 in BackendInitialize (port=0x561808e05700) at /home/rlott/git/stash/postgresql-fips/postgresql-9.3-9.3.10/build/../src/backend/postmaster/postmaster.c:4036
#10 BackendStartup (port=0x561808e05700) at /home/rlott/git/stash/postgresql-fips/postgresql-9.3-9.3.10/build/../src/backend/postmaster/postmaster.c:3807
#11 ServerLoop () at /home/rlott/git/stash/postgresql-fips/postgresql-9.3-9.3.10/build/../src/backend/postmaster/postmaster.c:1690
#12 0x00005618082cace1 in PostmasterMain (argc=5, argv=<optimized out>) at /home/rlott/git/stash/postgresql-fips/postgresql-9.3-9.3.10/build/../src/backend/postmaster/postmaster.c:1315
#13 0x0000561808103fb3 in main (argc=5, argv=0x561808db6970) at /home/rlott/git/stash/postgresql-fips/postgresql-9.3-9.3.10/build/../src/backend/main/main.c:227

I tracked it down to the following code in the OpenSSL 2.0.9 canister code:

int FIPS_drbg_generate(DRBG_CTX *dctx, unsigned char *out, size_t outlen,
int prediction_resistance,
const unsigned char *adin, size_t adinlen)
{
int r = 0;

if (FIPS_selftest_failed())
{
FIPSerr(FIPS_F_FIPS_DRBG_GENERATE, FIPS_R_SELFTEST_FAILED);
return 0;
}

if (!fips_drbg_check(dctx))
return 0;

if (dctx->status != DRBG_STATUS_READY
&& dctx->status != DRBG_STATUS_RESEED)
{
if (dctx->status == DRBG_STATUS_ERROR)
r = FIPS_R_IN_ERROR_STATE;
else if(dctx->status == DRBG_STATUS_UNINITIALISED)
r = FIPS_R_NOT_INSTANTIATED;
goto end;
}
...

The place where it fails is where dctx->status == DRBG_STATUS_UNINITIALIZED (i.e. 0).

So, my question is this: In FIPS mode, what would cause the random number generation to not initialize? I have put print statements in the postgresql code such that I know that it is in FIPS mode properly. I know that the dctx->status pointer, which points to a "static DRBG_CTX ossl_dctx" structure, is initialized to 1 in the main process. It appears that this initialization doesn't get propagated to other backends or the SSL transaction above.

If any of the developers have some insight into this, I would appreciate it.

Thanks,

Rodney Lott