Re: Rewriting the test of pg_upgrade as a TAP test - take two

On Fri, Mar 02, 2018 at 06:33:55PM +1030, Andrew Dunstan wrote:
> TestUpgradeXversion.pm is different in a number of respects from the
> inbuilt test regime. It doesn't run the normal regression suite to set
> up a test. Rather, it saves out the installed binaries and data
> directory of the buildfarm client, and then tries to upgrade those
> saved data directorories from earlier branches or the current branch
> to the target version being tested. It has to make a few adjustments
> to the databases along the way. It does a compare of pre- and post-
> pg_dumpall runs, allowing a fuzz factor if the versions are different.
> The other upside to the scheme is that we're testing pg_dump against
> earlier branches as part of testing pg_upgrade.

Thanks for the details. test.sh also does a comparison of the output
of pg_dumpall between the instance before and after upgrade. The
adjustments done are also close to what test.sh does, and what my patch
does. This consists in update prosrc for defined functions (+alpha),
right?

> I don't think a scheme like this is going to be terribly workable
> outside some system such as the buildfarm that deals with multiple
> branches.

Yeah, you need the code tree of the past instance as well as the
regression tests need to be run on the instance to upgrade.

> One of the significant pluses to TestUpgradeXversion.pm is that it
> tests upgrading quite a bit more than the standard regression
> database. It also tests all the contrib databases and the isolation
> and pl_regression databases. Several bugs have been found that way,
> IIRC, and we should arguably do something along the same lines for our
> builtin testing.

You have a point here. I did not consider those parts.

> I'll post a follow up when I've had a chance to have a good look at
> what Michael has actually done.

I was thinking about that a bit today, and a nice goal would be to come
up with an infrastructure that could be shared between the buildfarm and
the in-core code. Roughly I think that what I developed could be
changed so as we have TestUpgradeXversion.pm, which the buildfarm could
use, and the in-core TAP tests would be a thin wrapper on top of it.

I doubt also a bit how much people actually use test.sh to run
cross-version upgrades.. 5bab1985 is teaching this lesson.
--
Michael