Re: Review: extension template

Markus Wanner <markus(at)bluegap(dot)ch> writes:
>> Then what happens at pg_restore time? the CREATE EXTENSION in the backup
>> script will suddenly install the other extension's that happen to have
>> the same name? I think erroring out is the only safe way here.
>
> Extensions are commonly identified by name (installed ones as well as
> available ones, i.e. templates).
>
> Thus I think if a user renames a template, he might have good reasons to
> do so. He likely *wants* it to be a template for a different extension.
> Likewise when (re-)creating a template with the very same name of a
> pre-existing, installed extension.

I can understand that as a incomplete step towards a "migration" of some
sorts, but if we just allow to rename a template we open ourselves to be
producing non-restorable dumps (see below). I'm not at ease with that.

> Maybe the user just wanted to make a "backup" of the template prior to
> modifying it. If he then gives the new template the same name as the old
> one had, it very likely is similar, compatible or otherwise intended to
> replace the former one.
>
> The file-system templates work exactly that way (modulo DSOs). If you
> create an extension, then modify (or rename and re-create under the same
> name) the template on disk, then dump and restore, you end up with the
> new version of it. That's how it worked so far. It's simple to
> understand and use.

We have absolutely no control over the file-system templates and that's
why they work differently, I think. There's not even the notion of a
transaction over there.

> Or how do you think would pg_restore fail, if you followed the mental
> model of the template?

# create template for extension foo version 'x' as '';
# create extension foo;
# alter template for extension foo rename to bar;

$ pg_dump | psql

And now it's impossible to CREATE EXTENSION foo, because there's no
source to install it from available. I think we should actively prevent
that scenario to happen in the field (current patch prevents it).

Now, if I'm in the minority, let's just change that.

> However, this also means you restrict the user even further... How can
> he save a copy of an existing template, before (re-)creating it with
> CREATE TEMPLATE FOR EXTENSION?
>
> On the file system, a simple cp or mv is sufficient before
> (re)installing the package from your distribution, for example.

Usually what you do when you want to change an extension is that you
provide an upgrade script then run it with ALTER EXTENSION UPDATE.

Sometimes what you do is prepare a new installation script for a new
version of your extension and you don't provide an upgrade script, then
you update with the following method, in the case when you edited the
default_version property of the .control file:

# drop extension foo; -- drops version 1.0
# create extension foo; -- installs version 1.2

The current file system based extensions allow you to maintain
separately the files foo--1.0.sql and foo--1.2.sql, and you don't need
to copy a current version of the whole extension away before hacking the
new version.

The Extension Template facility in the current patch allows you to do
much the same:

# create template for extension foo version '1.0' as $foo$
create function foo() returns int language sql as $$ select 1; $$;
$foo$;

# create template for extension foo version '1.2' as $foo$
create function foo() returns int language sql as $$ select 2; $$;
$foo$;

# select ctlname, ctldefault, ctlversion
from pg_extension_control
where ctlname = 'foo';
ctlname | ctldefault | ctlversion
---------+------------+------------
foo | t | 1.0
foo | f | 1.2
(2 rows)

# create extension foo;
# select foo();
foo
-----
1
(1 row)

And now you can "upgrade" with:

# drop extension foo;
# create extension foo version '1.2';
# select foo();
foo
-----
2
(1 row)

Or even:

# alter template for extension foo set default version '1.2';
# drop extension foo;
# create extension foo;
# select foo();
foo
-----
2
(1 row)

So I don't see that we've broken any use case here, really. I think I
understand your objection in principles, but it appears to me that we
would gain nothing here by allowing broken pg_dump scripts.

> What way? And what community consensus?

I see that you've spent extra time and effort to better understand any
community consensus that might exist around this patch series, and I
want to say thank you for that!

> Re-reading some of the past discussions, I didn't find anybody voting
> for a dependency between the template and the created extension. And at
> least Tom pretty clearly had the template model in mind, when he wrote
> [1]: "We don't want it to look like manipulating a template has anything
> to do with altering an extension of the same name (which might or might
> not even be installed)." or [2]: "But conflating this functionality
> [i.e. extension templates] with installed extensions is just going to
> create headaches."

That was a comment against that proposal:

ALTER EXTENSION … CONFIGURATION FOR 'version' SET param = value, …;
ALTER EXTENSION … SET TEMPLATE FOR 'version' AS $$ … $$;
ALTER EXTENSION … SET TEMPLATE FROM 'version' TO 'version' AS …

So the patch we have now stays away from conflating templates and
extensions. Still, a dependency in between the extension object and the
template it came from is put into place so that we trust the pg_dump
script to contain enough information to be able to actually restore the
extension later. I think it's important to keep that dependency.

Basically what I'm saying in this too long email is that I need other
contributors to voice-in.

> The closest I found was Robert Haas mentioning [3], that "[he doesn't]
> see a problem having more than one kind of extensions". However, please
> mind the context. He doesn't really sound enthusiastic, either.

That was talking about a very different proposal than the one we're
currently analysing. The whole point of the current proposal has been to
avoid having another kind of extensions.

> I'm puzzled about some of your words in that thread. In the very message
> Robert responded to, you wrote [4]: "Having more than one way to ship an
> extension is good, having two different animals with two different
> incompatible behaviors named the same thing is bad."
>
> With the link-model, you are now proposing to create exactly that. Two
> different kinds of extensions that are not compatible with each other.
> One that is independent and one that depends on the "template" it got
> created from.

I don't see that, actually. The goal here still is that once installed
the behavior of an extension is always the same. The only difference we
have now is how to guarantee that we can restore a dump:

- using the filesystem based templates, no guarantee at all is offered
by PostgreSQL, that's the developer/sysadmin responsibility, the
dump script has external dependencies that you must manage yourself;

- using the catalog based templates, PostgreSQL guarantees that any
dump you take of the database will be self-contained wrt extensions,
so that you will be able to actually restore it.

Any other discrepency would be qualified as a bug I'm willing to fix.

> If that's really the case - which I doubt at the moment - I'll certainly
> accept that. I really recommend to rename the feature (and the
> commands), in that case, though. We may rather call the existing
> file-system thingie an "extension template", instead, as it becomes a
> good differentiator to what you're proposing.

Any proposals?

> How about ALTER EXTENSION ADD (or DROP)? With the link on the
> "template", you'd have to prohibit that ALTER as well, based on the
> exact same grounds as the RENAME: The installed extension would
> otherwise differ from the "template" it is linked to.

You're "supposed" to be using that from within the template scripts
themselves. The main use case is the upgrade scripts from "unpackaged".

I could see foreclosing that danger by enforcing that creating_extension
is true in those commands.

> See how this creates an animal pretty different from the current
> extensions? And IMO something that's needlessly restricted in many ways.

Well really I'm not convinced. If you use ALTER EXTENSION ADD against an
extension that you did install from the file system, then you don't know
what will happen after a dump and restore cycle, because you didn't
alter the files to match what you did, presumably.

If you do the same thing against an extension that you did install from
a catalog template, you just managed to open yourself to the same
hazards… I'd be already patching that like proposed above if it was not
for getting some more input and better assess if my understanding
matches that of Tom and Heikki on those fine points.

Regards,
--
Dimitri Fontaine
http://2ndQuadrant.fr PostgreSQL : Expertise, Formation et Support