test runner (was Re: SQL-standard function body)

Hi,

This started out as a reply to https://postgr.es/m/20210408170802.GA9392%40alvherre.pgsql
but it's independent enough to just start a new thread...

On 2021-04-08 13:08:02 -0400, Alvaro Herrera wrote:
> Yes, coverage.pg.org runs "make check-world".
>
> Maybe it would make sense to change that script, so that it runs the
> buildfarm's run_build.pl script instead of "make check-world". That
> would make coverage.pg.org report what the buildfarm actually tests ...
> it would have made this problem a bit more obvious.

We desperately need to unify the different test run environments we
have. I did spent some time trying to do that, and ended up with it
being hard to do in a good way in the make / msvc environment. Not sure
that I took the right path, but I end up doing experimental port of the
buildsystem meson - which has a builtin test runner (working on all
platforms...).

andres(at)awork3:/tmp$ ccache --clear
andres(at)awork3:/tmp$ ~/src/meson/meson.py setup ~/src/postgresql /tmp/pg-meson --prefix=/tmp/pg-meson-install
The Meson build system
Version: 0.57.999
Source dir: /home/andres/src/postgresql
Build dir: /tmp/pg-meson
Build type: native build
Project name: postgresql
Project version: 14devel
...
Header <unistd.h> has symbol "fdatasync" : YES
Header <fcntl.h> has symbol "F_FULLSYNC" : NO
Checking for alignment of "short" : 2
Checking for alignment of "int" : 4
...
Configuring pg_config_ext.h using configuration
Configuring pg_config.h using configuration
Configuring pg_config_paths.h using configuration
Program sed found: YES (/usr/bin/sed)
Build targets in project: 116

Found ninja-1.10.1 at /usr/bin/ninja
...

andres(at)awork3:/tmp/pg-meson$ time ninja
[10/1235] Generating snowball_create with a custom command
Generating tsearch script...............................
[41/1235] Generating generated_catalog_headers with a custom command
[1235/1235] Linking target contrib/test_decoding/test_decoding.so

real 0m10.752s
user 3m47.020s
sys 0m50.281s

...
andres(at)awork3:/tmp/pg-meson$ time ninja
[1/1] Generating test clean with a custom command

real 0m0.085s
user 0m0.068s
sys 0m0.016s
...

andres(at)awork3:/tmp/pg-meson$ time ~/src/meson/meson.py install --quiet
ninja: Entering directory `.'

real 0m0.541s
user 0m0.412s
sys 0m0.130s

...

andres(at)awork3:/tmp/pg-meson$ ninja test
[1/2] Running all tests.
1/74 postgresql:setup / temp_install OK 0.52s
2/74 postgresql:setup / cleanup_old OK 0.01s
3/74 postgresql:tap+pg_archivecleanup / pg_archivecleanup/t/010_pg_archivecleanup.pl OK 0.29s 42 subtests passed
4/74 postgresql:tap+pg_checksums / pg_checksums/t/001_basic.pl OK 0.27s 8 subtests passed
5/74 postgresql:tap+pg_config / pg_config/t/001_pg_config.pl OK 0.26s 20 subtests passed
...
68/74 postgresql:tap+pg_dump / pg_dump/t/002_pg_dump.pl OK 28.26s 6408 subtests passed
...
74/74 postgresql:isolation / pg_isolation_regress OK 114.91s

Ok: 74
Expected Fail: 0
Fail: 0
Unexpected Pass: 0
Skipped: 0
Timeout: 0

Full log written to /tmp/pg-meson/meson-logs/testlog.txt

And in cases of failures it'll show the failure when it happens
(including the command to rerun just that test, without the harness in
between), and then a summary at the end:

61/74 postgresql:tap+pg_verifybackup / pg_verifybackup/t/003_corruption.pl OK 10.65s 44 subtests passed
49/74 postgresql:tap+recovery / recovery/t/019_replslot_limit.pl ERROR 7.53s exit status 1
>>> MALLOC_PERTURB_=16 PATH=/tmp/pg-meson/tmp_install///usr/local/bin:/home/andres/bin/pg:/home/andres/bin/bin:/usr/sbin:/sbin:/home/andres/bin/pg:/home/andres/bin/bin:/usr/sbin:/sbin:/home/andres/bin/pg:/home/andres/bin/bin:/usr/sbin:/sbin:/home/andres/bin:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games:/snap/bin PG_REGRESS=/tmp/pg-meson/src/test/regress/pg_regress REGRESS_SHLIB=/tmp/pg-meson/src/test/regress/regress.so LD_LIBRARY_PATH=/tmp/pg-meson/tmp_install///usr/local/lib/x86_64-linux-gnu /home/andres/src/postgresql/src/tools/testwrap /tmp/pg-meson recovery t/019_replslot_limit.pl perl -I /home/andres/src/postgresql/src/test/perl -I /home/andres/src/postgresql/src/test/recovery /home/andres/src/postgresql/src/test/recovery/t/019_replslot_limit.pl
―――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――― ✀ ―――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――
stderr:
# Failed test 'check that required WAL segments are still available'
# at /home/andres/src/postgresql/src/test/recovery/t/019_replslot_limit.pl line 168.
# Looks like you failed 1 test of 14.

(test program exited with status code 1)
―――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――

62/74 postgresql:tap+pg_basebackup / pg_basebackup/t/010_dump_connstr.pl OK 11.59s 14 subtests passed
...
74/74 postgresql:isolation / pg_isolation_regress OK 112.26s

Summary of Failures:

49/74 postgresql:tap+recovery / recovery/t/019_replslot_limit.pl ERROR 7.53s exit status 1

Ok: 73
Expected Fail: 0
Fail: 1
Unexpected Pass: 0
Skipped: 0
Timeout: 0

Full log written to /tmp/pg-meson/meson-logs/testlog.txt
FAILED: meson-test
/usr/bin/python3 -u /home/andres/src/meson/meson.py test --no-rebuild --print-errorlogs
ninja: build stopped: subcommand failed.

It's hard to convey just how much nicer it is to see a progress report
during the test, see the failing tests at the end, without needing to
wade through reams of log output. The output of the individual tests is
in testlog.txt referenced above.

One can get a list of tests and then also just run subsets of them:

andres(at)awork3:/tmp/pg-meson$ ~/src/meson/meson.py test --list
postgresql:setup / temp_install
postgresql:setup / cleanup_old
postgresql:isolation / pg_isolation_regress
postgresql:regress / pg_regress
postgresql:tap+initdb / initdb/t/001_initdb.pl

Can run "suites" of tests:
~/src/meson/meson.py test --suite setup --suite recovery

Can run individual tests:
~/src/meson/meson.py test recovery/t/008_fsm_truncation.pl

Obviously all very far from being ready, but this seemed like a good
enough excuse to mention it ;)

Greetings,

Andres Freund