Re: Extension Templates S03E11

Hi,

Heikki Linnakangas <hlinnakangas(at)vmware(dot)com> writes:
> I still don't like this. What I suggested back in December was to have a
> simple mechanism to upload an extension zip file to the server via libpq
> (http://www.postgresql.org/message-id/50BF80A6.20500@vmware.com) The idea
> developed from that into the concept of extension templates, but the
> original idea was lost somewhere along the way.

And I will quote Andres' answer to your same proposal:

http://www.postgresql.org/message-id/20121205172747.GC27424@awork2.anarazel.de

So having a mode for pg_dump that actually makes dumps that are usable
for recovering after a disaster seems sensible to me. Otherwise you
need to redeploy from the VCS or whatever, which isn't really what you
want when restoring a database backup.

Comparing the situation to the one where you have extensions provided
by the packaging system or by /contrib or whatever doesn't seem to be
all that valid to me.

If you continue reading the thread from back then, the conclusion was to
drop the patch I was then proposing and instead work on the one we are
currently reviewing.

> Back in December, when I agreed that "upload zip file via libpq" might be
> useful, Tom suggested that we call control+sql file a "template", and the
> installed entity an "extension". So far so good. Now comes the patch, and
> the extension template no longer means a control+sql file. It means an
> entity that's installed in the database that contains the same information
> as a control+sql file, but in a new format. In fact, what *do* you call the
> control+sql file that lies on the filesystem? Not a template, apparently.

It's historical. To make it possible to start with some extension patch
in the 9.0 development cycle, it's been decided to only target the
contrib style extensions. Thanks to that we add something in 9.1.

In practice, the patch currently under review makes it so that both the
file system based model and the catalog based model behave the same (as
much as possible and sensible, and thanks to lots of reviewing efforts
from Markus Wanner), so we could be refering to the file system based
model as “template”.

> I want to be able to download extension.zip from pgxn.org, and then install
> it on a server. I want to be able to install it the traditional way, by
> unzipping it to the filesystem, or via libpq by using this new feature. I do
> *not* want to rewrite the extension using a new CREATE TEMPLATE FOR
> EXTENSION syntax to do the latter. I want to be able to install the *same*
> zip file using either method.

I would like to be able to support that, and the theory is attractive.
In practice, it's not that simple.

PGXN implements a source based distribution model, and most extensions
over there are following the same model, where the needed SQL files are
derived from sources at build time, using make(1).

See the following examples, the first one includes a C source file and
the second one is all PL stuff:

http://pgxn.org/dist/first_last_agg/
http://api.pgxn.org/src/first_last_agg/first_last_agg-0.1.2/

sql/$(EXTENSION)--$(EXTVERSION).sql: sql/$(EXTENSION).sql
cp $< $@

http://pgxn.org/dist/mimeo/1.0.1/
http://api.pgxn.org/src/mimeo/mimeo-1.0.1/

sql/$(EXTENSION)--$(EXTVERSION).sql: sql/tables/*.sql sql/functions/*.sql
cat $^ > $@

So, to support uploading PGXN zip files directly within the backend, now
the backend must be in a position to unpack the archive and build the
extension, then it must know where the build artefacts are going to be
found or it needs to `make install` in a known prefix and follow our
current conventions to find the files.

As I said to David Wheeler when he did build PGXN, I don't think that a
source level distribution is going to help us dealing with production
deployments.

So, while I understand where you're coming from, please tell me what are
your answers for those two design questions about the Extension template
idea:

- what should happen at pg_restore time?

- do you really want the extension build infrastructure in core?

My current thinking is to build the missing infrastructure as a
contrib module that will know how to divert CREATE EXTENSION with an
Event Trigger and apply the necessary magics at that time, and fill
in the Extension Templates for you.

Regards,
--
Dimitri Fontaine
http://2ndQuadrant.fr PostgreSQL : Expertise, Formation et Support