Re: RFC: Additional Directory for Extensions

On Nov 11, 2024, at 02:16, Peter Eisentraut <peter(at)eisentraut(dot)org> wrote:

> I implemented a patch along the lines Craig had suggested.

Oh, nice, thank you.

> It's a new GUC variable that is a path for extension control files. It's called extension_control_path, and it works exactly the same way as dynamic_library_path. Except that the magic token is called $system instead of $libdir.

I assume we can bikeshed these names later. :-)

> In fact, most of the patch is refactoring the routines in dfmgr.c to not hardcode dynamic_library_path but allow searching for any file in any path. Once a control file is found, the other extension support files (script files and auxiliary control files) are looked for in the same directory.

What about shared libraries files?

> This works pretty much fine for the use cases that have been presented here, including installing extensions outside of the core installation tree (for CNPG and Postgres.app) and for testing uninstalled extensions (for Debian).

If I understand correctly, shared modules still lie in dynamic_library_path, yes? That makes things a bit more complicated, as the CNPG use case has to set up multiple persistent volumes to persist files put into various directories, notably sharedir and pkglibdir.

> - You can install extensions into alternative directories using PGXS like
>
> make install datadir=/else/where/share pkglibdir=/else/where/lib
>
> This works. I was hoping it would work to use
>
> make install prefix=/else/where
>
> but that doesn't because of some details in Makefile.global. I think we can tweak that a little bit to make that work too.

In the broader extension organization RFC I’ve been working up[1], I propose a new Makefile prefix for the destination directory for an extension. It hinges on the idea that an extension has all of its files organized in a directory with the extension name, rather than separate params for data, pkglib, bin, etc.

> - With the current patch, if you install into datadir=/else/where/share, then you need to set extension_control_path=/else/where/share/extension. This is a bit confusing. Maybe we want to make the "extension" part implicit.

+1

> - The biggest problem is that many extensions set in their control file
>
> module_pathname = '$libdir/foo'
>
> This disables the use of dynamic_library_path, so this whole idea of installing an extension elsewhere won't work that way. The obvious solution is that extensions change this to just 'foo'. But this will require a lot updating work for many extensions, or a lot of patching by packagers.

Since that’s set at build/install time, couldn’t the definition of `$libdir` here be changed to mean “the directory into which it’s being installed right now?”. Doesn’t seem necessary to search a path if the specific location is set at install time.

> Maybe we could devise some sort of rule that if the extension control file is found via the path outside of $system, then the leading $libdir is ignored.
> <v0-0001-extension_control_path.patch>

This is what I propose,[1] yes. If we redefine the organization of extension files to live in a single directory named for an extension, then once you’ve found the control files all the other files are there, too. But `$libdir` is presumably still meaningful, since the first time you call an extension function in a new connection, it just tries to load that location, it doesn’t go through the control file search process, AFAIK.

Perhaps I misunderstand, but I would like to talk through the implications of a more radical rethinking of extension file location along the lines of the other thread[2] and the RFC I’m working up based on them both[1], especially since there are a few other use cases that inform it.

A notable one is the idea Gabriele shared with me in Athens to be able to add an extension to a running CNPG pod by simply mounting a read-only volume with all the files it requires.

Best,

David

[1]: https://github.com/theory/justatheory/pull/7/files