Re: TAP test breakage on MacOS X

On Sun, Oct 26, 2014 at 12:29 PM, Tom Lane <tgl(at)sss(dot)pgh(dot)pa(dot)us> wrote:
> The larger issue though is that even with both the above things fixed,
> the TAP tests would still be an expensive no-op on the majority of
> buildfarm members. AFAICT, I do not own a single machine on which the
> current TAP tests will consent to run, and in most cases that's after
> going out of my way to fetch CPAN modules that aren't in the vendor Perl
> installs. Peter's mostly been fixing the portability issues by disabling
> tests, which I guess is better than no fix at all, but it leaves darn
> little useful functionality.

I agree, emphatically. Honestly, if we can't get these tests running
everywhere with reasonable effort, we should just rip them out. We've
gone to a lot of trouble in general to make sure that our source code
can be ported even to systems that arguably nobody uses any more, and
instrumental to that effort is keeping the system requirements to
install and test PostgreSQL minimal. At this point, I wouldn't mind
moving the goalposts from C89 to C89 + a bunch of C99 features that
are available on all the platforms we have buildfarm coverage for, and
I wouldn't mind require perl to compile and install, full stop. But
this patch has gone much further than that: you need a new-enough
version of perl, and a new-enough version of a bunch of modules that
aren't installed by default, and maybe not even in the vendor install,
and the documentation on how to make it work is an embarrassment:

http://www.postgresql.org/docs/devel/static/regress-tap.html

Beyond all that, I have serious doubts about whether, even if we
eventually get these tests mostly working in most places, whether they
will actually catch any bugs. For example, consider dropdb. The TAP
tests cover the following points:

- When run with --help or --version, it should exit with code 0, print
something on stderr, and print nothing on stdout.
- When run with --not-a-valid-option, it should exit with a non-zero
exit code, print something on stderr, and print nothing on stdout.
- "dropdb foobar1" should cause "statement: DROP DATABASE foobar1" to
show up in the postgresql log file.
- "dropdb nonexistent" should exit non-zero.

These are certainly good things to test, but I'd argue that once
you've verified that they are working, they're unlikely to get broken
again in the future. I'm generally supportive of the idea that we
need more automated tests, but these tests seem pretty low-value to
me, even if they were running everywhere, and even moreso if they
aren't.

--
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company