How to manage shared library lifetime through C functions

Greetings,
I hope this is the right group to ask this question; apologies if this
should go the general or some other list.

I have multiple shared libraries that can be called from C that I'd like to
use from a C based postgresql function.

These libraries perform some expensive initialization and they require the
C code to properly release resources when the library is no longer needed.

This means that I need a mechanism to keep a session level pointer to a
library, initialize it when it is called first from a C based function and
dispose the library properly when the session ends (and terminated due to a
problem) I would like to keep the libraries available as long as the
session is alive, so multiple calls are supposed to avoid
initialization/disposal costs every time.

I could probably use a temp table as a container for the initalization and
even pointer values (sounds dirty) but I have no idea how to hook to
session end to clean up when session ends.

What would be a good strategy here?

Regards
Seref