Re: RFC: Extension Packaging & Lookup

Fellow Humans,

I’m working on an updated proposal with more detail, and more comprehensive. But I keep getting a bit caught up on this bit:

On Oct 28, 2024, at 18:19, David E. Wheeler <david(at)justatheory(dot)com> wrote:

>> * Binary-only extensions might also be installed here; the difference is they have no control file. The LOAD command and shared_preload_libraries would need to know to look here, too.
>
> Or perhaps we should require a control file for these, too, but add a “type” key or some such? Maybe such a shared module could be supported by CREATE EXTENSION, as well as, but not include SQL files?

I’m trying to imagine how this ought to work. The challenge is that, with the layout I propose here, shared module files will no longer always be in `$dynamic_library_path`, but in any `$extension/pkglib` subdirectory of each subdirectory of `extension_path`, as well. Is that desirable?

Let’s say we want to load a module named “semver” that’s included in the semver extension. With the proposed organization up-thread, the module would be installed in:

```
$extdir_user/semver/pkglib/semver.(so|dylib|dll|etc)
```

What should be passed to preload/LOAD to load it? A few options:

Option 1
--------

* Specify the module name “semver” in the `LOAD` command or in
`*_preload_libraries` (same as in 17 and earlier)
* Teach the preload/LOAD code to search for the module file in `*/pkglib/`
under each extension path

Pros:

* Follows the existing module name specification in preload/LOAD

Cons:

* Potentially huge number of directories to search, when lots of extension
are installed.
* Depending on search order, the wrong module may be loaded if two
extensions have a module file with the same name

Option 2
--------

* Specify the module name to include the extension name. Perhaps something
like `$extension:$module`.
* Teach the preload/LOAD code to detect the extension name as part of the
command and only look for the DSO in that extension's `pkglibdir`.

Pros:

* Searches at most the list of directories in the `extension_path`.
* No conflicts with any other module files from other extensions.

Cons:

* Overloads the meaning of preload/LOAD strings, which might be confusing to
some.
* Upgrades might need these values to change from the old to the new syntax.

Other Options?
--------------

I kind of like Option 2, as it would allow us to eventually support non-`CREATE EXTENSION` modules as extensions, too. I imagine distributing, say `auto_explain` in an extension directory of its own, with a `auto_explain.control` file that identifies it as a LOAD-only extension. Then specifying `auto_explain:auto_explain` would work as expected. Or perhaps just `auto_explain:` could load *all* the modules included in the auto_explain "extension".

But then maybe I'm starting to talk myself into arguing that `LOAD` ought to be deprecated, `CREATE EXTENSION` could support non-SQL extensions, and the `*preload*` GUCs would contain a list of extensions rather than module files.

But I digress. Any ideas about other options to address this design challenge?

Thanks,

David