Re: meson and check-tests

On Mon, Sep 23, 2024 at 2:16 PM Nazir Bilal Yavuz <byavuz81(at)gmail(dot)com> wrote:
>
> Hi,
>
> On Sat, 21 Sept 2024 at 09:01, jian he <jian(dot)universality(at)gmail(dot)com> wrote:
> >
> > hi. Thanks for your work!
>
> Thank you for looking into this!
>
> > I do find some issues.
> >
> >
> > TESTS="copy jsonb jsonb" meson test --suite regress
> > one will fail. not sure this is expected?
>
> Yes, that is expected.

Agreed. It's the same behaviour with `make check-tests`
TESTS="copy jsonb jsonb" make check-tests

# initializing database system by copying initdb template
# using temp instance on port 65312 with PID 880081
ok 1 - copy 51 ms
ok 2 - jsonb 143 ms
not ok 3 - jsonb 179 ms
1..3
# 1 of 3 tests failed.

>
> > so
> > TESTS="copy jsonb stats_ext" meson test --suite regress
> > will fail.
> >
> > to make it work we need change it to
> > TESTS="test_setup copy jsonb stats_ext" meson test --suite regress
> >
> > Many tests depend on test_setup.sql, maybe we can implicitly prepend it.
> > Another dependency issue. alter_table depending on create_index.
> >
> > TESTS="test_setup alter_table" meson test --suite regress
> > will fail.
> > TESTS="test_setup create_index alter_table" meson test --suite regress
> > will work.
>
> Yes, I realized that but since that is how it is done in the make
> builds, I didn't want to change the behaviour. Also, I think it makes
> sense to leave it to the tester. It is more flexible in that way.

Since meson has a setup suite already, it might have been a good idea
to do as Jian suggested. But a. setup is also another suite and not a
setup step per say. b. the dependencies between regression tests are
not documented well or rather we don't have a way to specify which
test depends upon which. So we can't infer the .sql files that need to
be run as a setup step. Somebody could add a dependency without meson
or make being aware of that and tests will fail again. So I think we
have to leave it as is. If we get to that point we should fix both
make as well as meson. But not as part of this exercise.

It's a bit inconvenient that we don't see whether an individual test
failed or succeeded on the screen; we need to open testlog.txt for the
same. But that's true with the regress suite generally so no
complaints there.

Individual TAP tests are run using `meson test -C <build dir>
<suite>:<test>` syntax. If we can run individual SQL tests using same
syntax e.g. `meson test regress:partition_join` that would make it
consistent with the way TAP tests are run. But we need to make sure
that the test later in the syntax would see the objects left behind by
prior tests. E.g. `meson test regress:test_setup
regress:partition_join` should see both tests passing. partition_join
uses some tables created by test_setup, so those need to be run
sequentially. Is that doable?

--
Best Wishes,
Ashutosh Bapat