Re: Another usability issue with our TAP tests

On Mon, Jul 16, 2018 at 01:13:36PM -0400, Tom Lane wrote:
> Since "make check-world" is rather chatty, if you get a failure while
> running it under high parallelism, the location of the failure has often
> scrolled off the terminal window by the time all the other subjobs
> exit.

Yes, I have pested about that myself a bit lately.

> This is not a huge problem for tests using our traditional infrastructure,
> because you can just run "git status" to look for regression.diffs files.
> But a TAP test failure leaves nothing behind that git will consider
> unusual. I've repeatedly had to run check-world with no parallelism
> (wasting many minutes) in order to locate which test actually failed.

Even for tests currently running regression.diffs is created but remains
empty.

> I'm not sure about a good way to improve this. One idea that comes
> to mind is to tweak the "make check" rules so that the tmp_check
> subdirectories are automatically deleted on successful completion,
> but not on failure, and then remove tmp_check from the .gitignore lists.
> But the trouble with that is sometimes you want to look at the test logs
> afterwards, even when make thought the test succeeded.

I definitely want to be able to look at TAP test logs even if they
succeed, and only wipe them out at the next run, so I think that this
should be independent. What about an on-disk empty file then which gets
simply created in the INIT() block of TestLib.pm, and removed in the END
block only if all tests pass? We know the base directory thanks to
$ENV{TESTDIR} which should just be "." if undefined. Then we name it
say, "prove_running" or similar.
--
Michael