tests and meson - test names and file locations

Hi,

One of the things motivating me to work on the meson conversion is the ability
to run tests in an easily understandable way. Meson has a testrunner that
both allows to run all tests at once, and run subsets of tests.

= Test and testsuite naming =

Each test has a unique name, and 0-n labels (called 'suites'). One can run
tests by their name, or select tests by suites.

The way I've organized it so far is that tests are named like this:

main/pg_regress
main/isolation
recovery/t/018_wal_optimize.pl
pg_prewarm/t/001_basic.pl
test_shm_mq/pg_regress
psql/t/001_basic.pl

we could also name tests by their full path, but that makes the display very
unwieldy.

At the moment there's three suites differentiating by the type of test:
'pg_regress', 'isolation' and 'tap'. There's also a separate "axis" of suites,
describing what's being tested, e.g. 'main', 'test_decoding', 'recovery' etc.

That currently works out to each test having two suites, although I've
wondered about adding a 'contrib' suite as well.

Perhaps the pg_regress suite should just be named 'regress'? And perhaps the
t/ in the tap tests is superfluous - the display is already fairly wide?

= Log and Data locations =

To make things like the selection of log files for a specific test easier,
I've so far set it up so that test data and logs are stored in a separate
directory from the sources.

testrun/<main|recovery|...>/<testname>/<log|tmp_check|results...>

The runner now creates a test.start at the start of a test and either
test.success or test.failure at the end. That should make it pretty easy for
e.g. the buildfarm and CI to make the logs for a failed test easily
accessible. I've spent far too much time going through the ~hundred logs in
src/test/recovery/ that the buildfarm displays as one thing.

I really like having all the test data separately from the sources, but I get
that that's not what we've done so far. It's doable to just mirror the current
choice, but I don't think we should. But I won't push too hard against keeping
things the same.

I do wonder if we should put test data and log files in a separate directory
tree, but that'd be a bit more work probably.

Any comments on the above?

= Example outputs =

Here's an example output that you mostly should be able to make sense of now:

$ m test --print-errorlogs
ninja: Entering directory `/tmp/meson'
ninja: no work to do.
1/242 postgresql:setup / tmp_install OK 0.33s
2/242 postgresql:tap+pg_upgrade / pg_upgrade/t/001_basic.pl OK 0.35s 8 subtests passed
3/242 postgresql:tap+recovery / recovery/t/011_crash_recovery.pl OK 2.67s 3 subtests passed
4/242 postgresql:tap+recovery / recovery/t/013_crash_restart.pl OK 2.91s 18 subtests passed
5/242 postgresql:tap+recovery / recovery/t/014_unlogged_reinit.pl OK 3.01s 23 subtests passed
6/242 postgresql:tap+recovery / recovery/t/022_crash_temp_files.pl OK 3.16s 11 subtests passed
7/242 postgresql:tap+recovery / recovery/t/016_min_consistency.pl OK 3.43s 1 subtests passed
8/242 postgresql:tap+recovery / recovery/t/021_row_visibility.pl OK 3.46s 10 subtests passed
9/242 postgresql:isolation+tcn / tcn/isolation OK 3.42s
10/242 postgresql:tap+recovery / recovery/t/023_pitr_prepared_xact.pl OK 3.63s 1 subtests passed
...
241/242 postgresql:isolation+main / main/isolation OK 46.69s
242/242 postgresql:tap+pg_upgrade / pg_upgrade/t/002_pg_upgrade.pl OK 57.00s 13 subtests passed

Ok: 242
Expected Fail: 0
Fail: 0
Unexpected Pass: 0
Skipped: 0
Timeout: 0

Full log written to /tmp/meson/meson-logs/testlog.txt

The 'postgresql' is because meson supports subprojects (both to provide
dependencies if needed, and "real" subprojects), and their tests can be run at
once.

If a test fails it'll show the error output at the time of test:

39/242 postgresql:pg_regress+cube / cube/pg_regress FAIL 3.74s exit status 1
>>> REGRESS_SHLIB=/tmp/meson/src/test/regress/regress.so MALLOC_PERTURB_=44 PG_TEST_EXTRA='kerberos ldap ssl' PG_REGRESS=/tmp/meson/src/test/regress/pg_regress PATH=/tmp/meson/tmp_install/tmp/meson-install/bin:/tmp/meson/contrib/cube:/home/andres/bin/perl5/bin:/home/andres/bin/pg:/home/andres/bin/bin:/usr/sbin:/sbin:/home/andres/bin/pg:/home/andres/bin/bin:/usr/sbin:/sbin:/usr/local/bin:/usr/bin:/bin:/usr/games /usr/bin/python3 /home/andres/src/postgresql/src/tools/testwrap --srcdir /home/andres/src/postgresql/contrib/cube --basedir /tmp/meson --builddir /tmp/meson/contrib/cube --testgroup cube --testname pg_regress /tmp/meson/src/test/regress/pg_regress --temp-instance /tmp/meson/testrun/cube/pg_regress/tmp_check --inputdir /home/andres/src/postgresql/contrib/cube --expecteddir /home/andres/src/postgresql/contrib/cube --outputdir /tmp/meson/testrun/cube/pg_regress --bindir '' --dlpath /tmp/meson/contrib/cube --max-concurrent-tests=20 --port=40012 cube cube_sci
――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――― ✀ ―――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――
# executing test in /tmp/meson/testrun/cube/pg_regress group cube test pg_regress, builddir /tmp/meson/contrib/cube
============== creating temporary instance ==============
============== initializing database system ==============
============== starting postmaster ==============
running on port 40012 with PID 354981
============== creating database "regression" ==============
CREATE DATABASE
ALTER DATABASE
ALTER DATABASE
ALTER DATABASE
ALTER DATABASE
ALTER DATABASE
ALTER DATABASE
============== running regression test queries ==============
test cube ... FAILED 418 ms
test cube_sci ... ok 16 ms
============== shutting down postmaster ==============

======================
1 of 2 tests failed.
======================

The differences that caused some tests to fail can be viewed in the
file "/tmp/meson/testrun/cube/pg_regress/regression.diffs". A copy of the test summary that you see
above is saved in the file "/tmp/meson/testrun/cube/pg_regress/regression.out".

# test failed
――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――

40/242 postgresql:pg_regress+citext / citext/pg_regress OK 3.77s
...

A list of failed tests is listed at the end of the test:

Summary of Failures:

39/242 postgresql:pg_regress+cube / cube/pg_regress FAIL 3.74s exit status 1

Ok: 241
Expected Fail: 0
Fail: 1
Unexpected Pass: 0
Skipped: 0
Timeout: 0

Full log written to /tmp/meson/meson-logs/testlog.txt

Greetings,

Andres Freund