set TESTDIR from perl rather than Makefile

Forking: <E6CA9D45-96EA-4037-8BEB-344CCA75A38C(at)anarazel(dot)de>

On Tue, Feb 15, 2022 at 10:42:09PM -0800, Andres Freund wrote:
> >> I was thinking that we should make Utils.pm's INIT block responsible for
> >> figuring out both the directory a test should run in and the log location,
> >> instead having that in vcregress.pl and Makefile.global.in. Mostly because
> >> doing it in the latter means we can't start tests with different TESTDIR and
> >> working dir at the same time.
> >>
> >> If instead we pass the location of the top-level build and top-level source
> >> directory from vcregress.pl / Makefile.global, the tap test infrastructure can
> >> figure out that stuff themselves, on a per-test basis.
> >>
> >> For msvc builds we probably would need to pass in some information that allow
> >> Utils.pm to set up PATH appropriately. I think that might just require knowing
> >> that a) msvc build system is used b) Release vs Debug.
> >
> >I'm totally unsure if this resembles what you're thinking of, and I'm surprised
> >I got it working so easily. But it gets the tap test output in separate dirs,
> >and CI is passing for everyone (windows failed because I injected a "false" to
> >force it to upload artifacts).
> >
> >https://github.com/justinpryzby/postgres/runs/5211673291
>
> Yes, that's along the lines I was thinking. I only checked it on my phone, so it certainly isn't a careful look...
>
> I think this should be discussed in a separate thread, for visibility.

I rebased and fixed the check-guc script to work, made it work with vpath
builds, and cleaned it up some.

There may be other reasons to do this, but the reason I did it is to implement
an alltaptests target for vcregress, for cirrus (and everyone else). If all
the tap tests are run serially, it takes ~16min on cirrus; it takes ~13 to run
in parallel (the below run is slower than that since all the slow tests were
scheduled at once - which isn't always a good idea).

Running the tests in parallel uses a single invocation of prove, and starts all
the TAP tests from the same dir rather than their own dir. But without this
patch, all the tap test are output to the same dir, which is a pain to look
through.

This (and other) patches ran here.
https://github.com/justinpryzby/postgres/runs/5261323874
...
e806bcb280 wip: set TESTDIR from src/test/perl rather than Makefile/vcregress
a1bfa8e1a6 cirrus/windows: increase timout to 20min
5479b44198 vcregress: add alltaptests
fcef696c7d vcregress: run alltaptests in parallel
2d7dba13dd cirrus/windows: prove --state to run tests in order
7edf835d43 tmp: run tap tests first
6fb010c137 cirrus: include hints how to install OS packages..
28a25f12c3 cirrus/windows: add compiler_warnings_script
75bc8cff69 cirrus: upload changed html docs as artifacts
8008af2480 s!build docs as a separate task..
a3bf699a0e vcregress/ci: test modules/contrib with NO_INSTALLCHECK=1
d28ce46c2f wip: cirrus: code coverage

--
Justin