Re: Testing Extension Upgrade Paths

On 12 December 2017 at 07:25, Michael Paquier <michael(dot)paquier(at)gmail(dot)com>
wrote:

> On Tue, Dec 12, 2017 at 2:55 AM, Jeremy Finzel <finzelj(at)gmail(dot)com> wrote:
> > I understand how to setup a regression suite for a postgres extension,
> but
> > what I'm not clear on from the docs is if there is a pattern that exists
> for
> > testing not only the latest version of an extension, but also an upgraded
> > previous version.
> >
> > Is there any straightforward way to do this that doesn't involve manually
> > copying tests?
>
> It is perfectly possible to do tests on a specific version and test
> upgrade paths using CREATE EXTENSION and ALTER EXTENSION which support
> versioning in a SQL regression script, say:
> CREATE EXTENSION foo VERSION "0.1";
> -- Do stuff
> ALTER EXTENSION foo UPDATE TO "0.2";
> -- Do other stuff
>

This works well when you want to test 1.1 binaries with a 1.0 SQL extension
definition.

It's not effective if you need to test 1.0 binaries+extension, then an
upgrade to 1.1 binaries and extension. You have no way to install and run
the 1.0 binaries without going outside pg_regress / TAP and using an
external test harness/framework/script. I've been bitten by this multiple
times before. For example, if 1.1 always assigns a value to some column
that was nullable in 1.0, you're never going to exercise 1.1's handling of
nulls in that column.

It also doesn't lend its self to complex multi-path upgrades, where for
example you could upgrade 1.0.0 -> 1.0.1 -> 1.1.0 or upgrade directly from
1.0.0 -> 1.1.0. This rapidly becomes an issue when you release 1.0.0,
release 1.1.0, then release a maintenance 1.0.1 version. Now you have
toadd the 1.0.1 -> 1.1.0 upgrade path, but you still have the 1.0.0 ->
1.1.0 path.

You can handle that with TAP if you're willing to write something to do the
various combinations of steps and exercise them. It's not practical with
pg_regress.

More thorough testing benefits from an external harness.

--
Craig Ringer http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Training & Services