Re: TAP test breakage on MacOS X

On 10/27/2014 11:41 AM, Robert Haas wrote:
> On Sun, Oct 26, 2014 at 12:29 PM, Tom Lane <tgl(at)sss(dot)pgh(dot)pa(dot)us> wrote:
>> The larger issue though is that even with both the above things fixed,
>> the TAP tests would still be an expensive no-op on the majority of
>> buildfarm members. AFAICT, I do not own a single machine on which the
>> current TAP tests will consent to run, and in most cases that's after
>> going out of my way to fetch CPAN modules that aren't in the vendor Perl
>> installs. Peter's mostly been fixing the portability issues by disabling
>> tests, which I guess is better than no fix at all, but it leaves darn
>> little useful functionality.
> I agree, emphatically. Honestly, if we can't get these tests running
> everywhere with reasonable effort, we should just rip them out. We've
> gone to a lot of trouble in general to make sure that our source code
> can be ported even to systems that arguably nobody uses any more, and
> instrumental to that effort is keeping the system requirements to
> install and test PostgreSQL minimal. At this point, I wouldn't mind
> moving the goalposts from C89 to C89 + a bunch of C99 features that
> are available on all the platforms we have buildfarm coverage for, and
> I wouldn't mind require perl to compile and install, full stop. But
> this patch has gone much further than that: you need a new-enough
> version of perl, and a new-enough version of a bunch of modules that
> aren't installed by default, and maybe not even in the vendor install,
> and the documentation on how to make it work is an embarrassment:
>
> http://www.postgresql.org/docs/devel/static/regress-tap.html
>
> Beyond all that, I have serious doubts about whether, even if we
> eventually get these tests mostly working in most places, whether they
> will actually catch any bugs. For example, consider dropdb. The TAP
> tests cover the following points:
>
> - When run with --help or --version, it should exit with code 0, print
> something on stderr, and print nothing on stdout.
> - When run with --not-a-valid-option, it should exit with a non-zero
> exit code, print something on stderr, and print nothing on stdout.
> - "dropdb foobar1" should cause "statement: DROP DATABASE foobar1" to
> show up in the postgresql log file.
> - "dropdb nonexistent" should exit non-zero.
>
> These are certainly good things to test, but I'd argue that once
> you've verified that they are working, they're unlikely to get broken
> again in the future. I'm generally supportive of the idea that we
> need more automated tests, but these tests seem pretty low-value to
> me, even if they were running everywhere, and even moreso if they
> aren't.
>

Yeah. I think at the very least they should be removed from the
check-world and installcheck-world targets until this is sorted out.

cheers

andrew