Re: pg_upgrade automatic testing

On sön, 2011-11-27 at 19:12 -0500, Andrew Dunstan wrote:
> Contrib tests are only run by the buildfarm in installcheck mode, so
> that will probably turn the tests off for the buildfarm, so +1 on that

Does the new buildfarm modular thing support that some members could run
the checks through explicit configuration?

> I think these tests are probably somewhat ill-conceived. Note also
> that shell scripts are not portable, and so these tests would not be
> able to run on MSVC buildfarm members, even if they had been enabled in
> the MSVC regression driver, which they have not. If we need a regression
> driver script it needs to be written in Perl.

Anyone is free to rewrite the thing in a different language.

> Also note that the test as written is likely to add significantly to
> buildfarm run times, as it will run the entire base regression suite
> again, possibly several times.

Are there any restrictions on how long a buildfarm run is supposed to
take?

> Finally, I think that this is probably not what we really need.

What do you base your thinking on?

This test suite has already found a number of bugs in the pg_upgrade
procedure that no one else was able to find. By that measure, it's
exactly what we need.

> I have
> already started work (as I mentioned some weeks ago) on having the
> buildfarm stash away a successful build and data directory, to be used
> later in cross-version upgrade testing, which seems to me much more what
> we need from something like the buildfarm. Maybe we could discuss how to
> run something like that.

That is one part of the puzzle. But once you have stashed away the old
source and data directory, you still need a test runner, which is
exactly what this provides you.

But note, cross-version pg_upgrade checks will not give you the full
value, even assuming that you can make them work at all in an unattended
way, because by default you won't be able to get a clean "dumps match"
result, at least without a lot of additional work to mangle the dump
output. Most (or all) of the bugs found so far with this test suite
were for upgrades *from* whatever was the current version. If we don't
have a current-to-current upgrade test suite, then we would only find
those years from now.