Re: RFC: Additional Directory for Extensions

Hi everyone,

I have finally been able to test the patch in a Kubernetes environment with
CloudNativePG, thanks to Niccolò Fei and Marco Nenciarini, who created a
pilot patch for CloudNativePG (
https://github.com/cloudnative-pg/cloudnative-pg/pull/6546)

In the meantime, Kubernetes is likely adding the ImageVolume feature
starting from the upcoming version 1.33. I will write a blog post soon
about how CloudNativePG will benefit from this feature. See
https://github.com/kubernetes/enhancements/issues/4639.

Although the steps are not easily reproducible by everyone, I can confirm
that I successfully mounted a volume in the Postgres pod using a container
image that includes only pgvector (with a size of 1.6MB - see
https://github.com/EnterpriseDB/pgvector/blob/dev/5645/Dockerfile.cnpg)

By setting:

dynamic_library_path = '$libdir:/extensions/pgvector/lib'
extension_control_path = '$system:/extensions/pgvector/share'

I was able to run the following queries:

postgres=# SELECT * FROM pg_available_extensions WHERE name = 'vector';
-[ RECORD 1 ]-----+-----------------------------------------------------
name | vector
default_version | 0.8.0
installed_version |
comment | vector data type and ivfflat and hnsw access methods

postgres=# SELECT * FROM pg_available_extensions WHERE name = 'vector';
-[ RECORD 1 ]-----+-----------------------------------------------------
name | vector
default_version | 0.8.0
installed_version |
comment | vector data type and ivfflat and hnsw access methods

I also successfully ran the following:

postgres=# SELECT * FROM pg_extension_update_paths('vector');

By emptying the content of `extension_control_path`, the vector extension
disappeared from the list.

postgres=# SHOW extension_control_path ;
extension_control_path
------------------------
$system
(1 row)

postgres=# SELECT * FROM pg_available_extensions WHERE name = 'vector';
name | default_version | installed_version | comment
------+-----------------+-------------------+---------
(0 rows)

postgres=# SELECT * FROM pg_available_extension_versions WHERE name =
'vector';
name | version | installed | superuser | trusted | relocatable | schema |
requires | comment
------+---------+-----------+-----------+---------+-------------+--------+----------+---------
(0 rows)

In my opinion, the patch already helps a lot and does what I can reasonably
expect from a first iteration of improvements in enabling immutable
container images that ship a self-contained extension to be dynamically
loaded and unloaded from a Postgres cluster in Kubernetes. From here, it is
all about learning how to improve things with an exploratory mindset with
future versions of Postgres and Kubernetes. It's a long journey but this is
a fundamental step in the right direction.

Let me know if there's more testing for me to do. The documentation looks
clear to me.

Thank you to everyone who contributed to this patch, from the initial
discussions to the development phase. I sincerely hope this is included in
Postgres 18.

Ciao,
Gabriele

On Fri, 28 Feb 2025 at 16:36, Matheus Alcantara <matheusssilv97(at)gmail(dot)com>
wrote:

> Hi
>
> On Tue, Feb 25, 2025 at 5:29 PM Matheus Alcantara
> <matheusssilv97(at)gmail(dot)com> wrote:
> > Fixed on the attached v3.
> >
> After I've sent the v3 patch I noticed that the tests were failing on
> windows.
> The problem was on TAP test that was using ":" as a separator on
> extension_control_path and also the path was not being escaped correctly
> resulting in a wrong path configuration.
>
> Attached v4 with all these fixes.
>
> --
> Matheus Alcantara
>

--
Gabriele Bartolini
VP, Chief Architect, Kubernetes
enterprisedb.com