pgcrypto: better memory management?...

I was working on a crypto extension many, many years ago but...life. And I
found the book "Encryption in the Database" which made me rethink many
things. (It describes the approach used by Oracle.)

But between openssl 3 and some other tasks I've been revisiting some of
these ideas and focused on the pgcrypto extension first.

Something that immediately struck me is memory management. There's two
items.

The first is that openssl has supported custom memory management for a very
long time - I remember using it in the late 90s. The function is in crypto.h

typedef void *(*CRYPTO_malloc_fn)(size_t num, const char *file, int line);
typedef void *(*CRYPTO_realloc_fn)(void *addr, size_t num, const char
*file,
int line);
typedef void (*CRYPTO_free_fn)(void *addr, const char *file, int line);
int CRYPTO_set_mem_functions(CRYPTO_malloc_fn malloc_fn,
CRYPTO_realloc_fn realloc_fn,
CRYPTO_free_fn free_fn);

The main benefit of this function is that it openssl-internal functions
will use the same memory pool as your application. I know the current
extension uses palloc/pfree but I didn't see a call to this function.

I didn't have any problems with my extension but I never put the server +
extension under a heavy load. There may be problems with this approach -
but I think the benefits of a single memory pool are large enough to
warrant investigating this.

The second item is that openssl also has these wonderful functions:

int CRYPTO_secure_malloc_init(size_t sz, size_t minsize);
int CRYPTO_secure_malloc_done(void);
void *CRYPTO_secure_malloc(size_t num, const char *file, int line);
void *CRYPTO_secure_zalloc(size_t num, const char *file, int line);
void CRYPTO_secure_free(void *ptr, const char *file, int line);
void CRYPTO_secure_clear_free(void *ptr, size_t num,
const char *file, int line);
int CRYPTO_secure_allocated(const void *ptr);
int CRYPTO_secure_malloc_initialized(void);
size_t CRYPTO_secure_actual_size(void *ptr);
size_t CRYPTO_secure_used(void);

void OPENSSL_cleanse(void *ptr, size_t len);

These functions address (at least) two potential vulnerabilities. First it
tweaks the memory block so it will never be written to the swap file, and
second it (iirc) also tweaks the memory block to reduce the access
permissions further than usual.

These functions should only be used for sensitive values, e.g., unprotected
keys. There's also the usual advice to minimize the amount of time the
sensitive data is kept in memory - do a clear + free immediately after use,
not as the last thing you do before your function returns.

It's important to remember that this memory pool is managed separately from
the rest. It has to be - it has to be able to manipulate the mmap flags.

Of course we can use a conditional #define to define these functions if the
user is running an old version of libssl. I would have to research when
they were introduced.

Broader applicability

I want to point out that the latter functions may be useful in other parts
of the system. E.g., the client session may have sensitive information that
should be as protected as possible. However that may not be possible since
it would require the server, not just the extension, to load and initialize
libssl

Sample code

I'll create my own github fork so it will be easy to create pull requests
for further discussion.

Bear