Re: TAP output format in pg_regress

Attached is a new version of the pg_regress TAP patch which - per reviewer
feedback - does away with dual output formats and just converts the existing
output to be TAP complaint. The idea was to keep it fairly human readable
while still be TAP compliant enough that running it with prove (with a tiny
Perl wrapper) works. This version also strips away the verbose output which
these days isn't terribly useful and mainly consume vertical space. Another
feature of the patch is to switch error logging to using the common frontend
logging rather than pg_regress rolling its own. The output from pg_log_error
is emitted verbatim by prove as it's on stderr.

I based the support on the original TAP specification and not the new v13 or
v14 since it seemed unclear how well those are supported in testrunners. If
v14 is adopted by testrunners there are some better functionality for reporting
errors that we could use, but for how this version seems a safer option.

A normal "make check" with this patch applied now looks like this:

+++ regress check in src/test/regress +++
# running on port 61696 with PID 57910
ok 1 - test_setup 326 ms
ok 2 - tablespace 401 ms
# parallel group (20 tests): varchar char oid pg_lsn txid int4 regproc money int2 uuid float4 text name boolean int8 enum float8 bit numeric rangetypes
ok 3 - boolean 129 ms
ok 4 - char 73 ms
ok 5 - name 117 ms
ok 6 - varchar 68 ms
<...snip...>
ok 210 - memoize 137 ms
ok 211 - stats 851 ms
# parallel group (2 tests): event_trigger oidjoins
ok 212 - event_trigger 138 ms
not ok 213 - oidjoins 190 ms
ok 214 - fast_default 149 ms
1..214
# 1 of 214 tests failed.
# The differences that caused some tests to fail can be viewed in the
# file "/<path>/src/test/regress/regression.diffs". A copy of the test summary that you see
# above is saved in the file "/<path>/src/test/regress/regression.out".

The ending comment isn't currently shown when executed via prove as it has to
go to STDERR for that to work, and I'm not sure that's something we want to do
in the general case. I don't expect running the pg_regress tests via prove is
going to be the common case. How does the meson TAP ingestion handle
stdout/stderr?

And for the sake of completeness, even though we all know this by heart, a
similar run from the current output format looks like:

+++ regress check in src/test/regress +++
============== creating temporary instance ==============
============== initializing database system ==============
============== starting postmaster ==============
running on port 61696 with PID 61955
============== creating database "regression" ==============
CREATE DATABASE
ALTER DATABASE
ALTER DATABASE
ALTER DATABASE
ALTER DATABASE
ALTER DATABASE
ALTER DATABASE
============== running regression test queries ==============
test test_setup ... ok 469 ms
test tablespace ... ok 415 ms
parallel group (20 tests): varchar char oid name int2 pg_lsn int4 txid text uuid regproc boolean money float4 int8 float8 bit enum numeric rangetypes
boolean ... ok 105 ms
char ... ok 64 ms
name ... ok 70 ms
varchar ... ok 55 ms
<...snip...>
memoize ... ok 149 ms
stats ... ok 873 ms
parallel group (2 tests): event_trigger oidjoins
event_trigger ... FAILED 142 ms
oidjoins ... FAILED 208 ms
test fast_default ... ok 172 ms
============== shutting down postmaster ==============

========================
2 of 214 tests failed.
========================

The differences that caused some tests to fail can be viewed in the
file "/<path>/src/test/regress/regression.diffs". A copy of the test summary that you see
above is saved in the file "/<path>/src/test/regress/regression.out".

There is a slight reduction in information, for example around tests run
serially vs in a parallel group. This could perhaps be handled by marking the
test name in some way like "tablespace (serial) or prefixing with a symbol or
somerthing. I can take a stab at that in case we think that level of detail is
important to preserve.

--
Daniel Gustafsson https://vmware.com/