Re: TAP test breakage on MacOS X

On 10/27/14 11:41 AM, Robert Haas wrote:
> Beyond all that, I have serious doubts about whether, even if we
> eventually get these tests mostly working in most places, whether they
> will actually catch any bugs.

Well, they caught the fact that pg_basebackup can't back up tablespaces
with names longer than 99 characters, for example.

But it's wrong to expect the primary value of tests to be to detect
previously unknown bugs. Yes, that has been the experience in this
project. We have software that we think works, and then we send it out
to test on N obscure platforms, and we find interesting things.

But the other dimension is that tests allow you to make changes with
confidence. For example, the tablespace mapping logic in pg_basebackup
has been whacked around about three times. I don't have any doubt that
it still works correctly, because it has extensive test coverage. If
someone sends in a patch to support cpio in pg_basebackup (hey, no
symlink name length limits) and wants to refactor the entire source code
to achieve that, I'll have no problem with that.

Yes, some of the tests are low in value, perhaps a bit silly. But it's
a foundation, and I expect to add more in the future. The good news is
that because some of these "silly" tests actually exercise important
internal functionality of the test mechanisms (e.g., create temporary
directory, run program, record exit status, capture stdout and stderr
separately, check output files), I am pretty confident that we can go
far with the current infrastructure without needing any more external
modules or something like that.

So that also means that not all tests need to be run everywhere all the
time. We have the option to run specific tests. It could be useful to
have better groupings, but that has been discussed before and it hasn't
come to a solution.

> These are certainly good things to test, but I'd argue that once
> you've verified that they are working, they're unlikely to get broken
> again in the future.

Famous last words ... ;-)