Re: RFC: Extension Packaging & Lookup

Thanks for this, David,

> On Oct 28, 2024, at 3:19 PM, David E. Wheeler <david(at)justatheory(dot)com> wrote:
>
> ## Challenge: Third Party Dependencies

This of course is the area that worries the heck out of me, as someone with extensions that includes not just single system dependencies but long chains of them (depending on GDAL draws in a huge tree).
>
> For solutions that require installing a DSO outside the default directories that the system is aware of, one needs a way to tell the system where to find them. There are two basic methods for doing so:
>
> 1. Set LD_LIBRARY_PATH to the directory in which third-party DSOs are installed. Today, however, this is considered an insecure pattern[7]. It doesn’t work at all on macOS, for example, unless you disable SIP[8]. Few will do so, nor should they. Andres Freund reports that it’s on its way out on Linux, too. So perhaps some can do this, but it sounds as if LD_LIBRARY_PATH’s days are numbered.
>
> 2. Use `-rpath` when compiling the DSOs to point to the proper place. This embeds the path in the DSO, so it always looks in the same place. However, this requires that the DSO be recompiled for every variant of the `-rpath`, which creates challenges for non-path specific binary packaging --- or if an OS vendor changes the director. But perhaps Postgres itself could be compiled with an `-rpath` that will be used when loading extensions, so the extension DSOs themselves don’t have to know the path? Then the base image just needs to be compiled with that option.

I’m unsure if it will work, but I have wondered if building out the dependencies to install right next to the DSO, and giving the DSO an rpath of “.” would achieve the effect we are looking for. It’s unfortunate (DY)LD_LIBRARY_PATH is dead and dying, but there we are. The trouble I see with somehow coercing the system to load a local copy of system libraries is for (a) common system libs that PostgreSQL itself might be linking (libssl, for example) that then will end up with symbol collisions between the copy loaded by postgres and the copy loaded by the DSO and (b) same thing but for different extensions with the same dependencies.

I guess I cannot shake the idea that a lot of interesting extensions are going to have interesting system dependencies, that “exposing an interesting library to postgres” has a high value for an integration system like PostgreSQL.

Question for the more knowledgable, how are binary distribution systems like Conda and others shipping DLLs such that different packages don’t clobber each other?

P.

>
> ## Comments and Corrections
>
> I think I captured most of the issues we discussed at PGConf.eu <http://pgconf.eu/> last week; please correct any misunderstandings, inaccuracies, and oversights you spot! And are there any other issues you can think of with the overall approach?
>
> Thanks for reading to the end!
>
> Best,
>
> David
>
> [1]: https://www.postgresql.org/message-id/D30A91FA-A6D4-4737-941F-0BBB2984B730%40justatheory.com
> [2]: https://github.com/ossc-db/pg_hint_plan/
> [3]: https://commitfest.postgresql.org/50/4913/
> [4]: https://superuser.com/a/1603150/285886
> [5]: https://github.com/pramsey/pgsql-http
> [6]: https://stackoverflow.com/a/42260979/79202
> [7]: http://xahlee.info/UnixResource_dir/_/ldpath.html
> [8]: https://developer.apple.com/documentation/security/disabling-and-enabling-system-integrity-protection
>
>
>