Re: RFC: Extension Packaging & Lookup

> On Oct 29, 2024, at 10:09 AM, David E. Wheeler <david(at)justatheory(dot)com> wrote:
>
> On Oct 29, 2024, at 12:23, Paul Ramsey <pramsey(at)cleverelephant(dot)ca> wrote:
>
>> Thanks for this, David,
>
> 🤘🏻
>
>> This of course is the area that worries the heck out of me, as someone with extensions that includes not just single system dependencies but long chains of them (depending on GDAL draws in a huge tree).
>
> Yeah. I cited pgsql-http as a simple place to start, on the assumption that once we figure out how to properly configure things for one DSO, it the pattern should work for any of them in a tree.

An apposite choice, since it not only demonstrates depending on a common system library, it also demonstrates the way these things loop on each other, as curl then depends on libssl, which postgres also depends on.

> I’m unsure if it will work, but I have wondered if building out the dependencies to install right next to the DSO, and giving the DSO an rpath of “.” would achieve the effect we are looking for.
>
> Given the security issues with library paths, I’d guess that relative paths are verboten. But also, Postgres does not `cd` into an extension directory before loading it, AFAIK.

Relative rpaths as I have seen them are relative to the executable or library in which they are defined (as far as I know, I’m not a dylib expert by any stretch). The implication is that extension.so <http://extension.so/> could have an rpath=. and dependent dylibs sitting next to it. This is how, for example, cmake out-of-tree builds can run tests against the newly built library before it’s installed,.. the test execs have an rpath of ../lib on them.

>
>> It’s unfortunate (DY)LD_LIBRARY_PATH is dead and dying, but there we are. The trouble I see with somehow coercing the system to load a local copy of system libraries is for (a) common system libs that PostgreSQL itself might be linking (libssl, for example) that then will end up with symbol collisions between the copy loaded by postgres and the copy loaded by the DSO and (b) same thing but for different extensions with the same dependencies.
>
> Yeah, this is why people tend to depend on system dependencies loaded from well-known paths, so libssl will always load the same DSO. I imagine the use of LD_LIBRARY_PATH can cause issues today.
>
>> I guess I cannot shake the idea that a lot of interesting extensions are going to have interesting system dependencies, that “exposing an interesting library to postgres” has a high value for an integration system like PostgreSQL.
>
> Yeah, I think the issue will be to figure out how to manage OS package-provided system dependencies in immutable environments like a Docker container. I suspect some combination of -rpath compiled into Postgres and mounting individual DSO files not included in the base image will be the way to go.

Do you see immutable images as the main deployment path going forward? I’m not container-pilled yet, so it still feels like a niche concern. Meanwhile being able to “add an extension that my cloud provider doesn’t support yet” feels quite vital.

ATB,

P