Re: meson and check-tests

Hi,

Thanks for looking into this!

On Wed, 25 Sept 2024 at 13:27, Ashutosh Bapat
<ashutosh(dot)bapat(dot)oss(at)gmail(dot)com> wrote:
>
> On Mon, Sep 23, 2024 at 2:16 PM Nazir Bilal Yavuz <byavuz81(at)gmail(dot)com> wrote:
> >
> > On Sat, 21 Sept 2024 at 09:01, jian he <jian(dot)universality(at)gmail(dot)com> wrote:
> > >
> > > so
> > > TESTS="copy jsonb stats_ext" meson test --suite regress
> > > will fail.
> > >
> > > to make it work we need change it to
> > > TESTS="test_setup copy jsonb stats_ext" meson test --suite regress
> > >
> > > Many tests depend on test_setup.sql, maybe we can implicitly prepend it.
> > > Another dependency issue. alter_table depending on create_index.
> > >
> > > TESTS="test_setup alter_table" meson test --suite regress
> > > will fail.
> > > TESTS="test_setup create_index alter_table" meson test --suite regress
> > > will work.
> >
> > Yes, I realized that but since that is how it is done in the make
> > builds, I didn't want to change the behaviour. Also, I think it makes
> > sense to leave it to the tester. It is more flexible in that way.
>
> Since meson has a setup suite already, it might have been a good idea
> to do as Jian suggested. But a. setup is also another suite and not a
> setup step per say. b. the dependencies between regression tests are
> not documented well or rather we don't have a way to specify which
> test depends upon which. So we can't infer the .sql files that need to
> be run as a setup step. Somebody could add a dependency without meson
> or make being aware of that and tests will fail again. So I think we
> have to leave it as is. If we get to that point we should fix both
> make as well as meson. But not as part of this exercise.
>
> It's a bit inconvenient that we don't see whether an individual test
> failed or succeeded on the screen; we need to open testlog.txt for the
> same. But that's true with the regress suite generally so no
> complaints there.

Thanks for sharing your thoughts.

> Individual TAP tests are run using `meson test -C <build dir>
> <suite>:<test>` syntax. If we can run individual SQL tests using same
> syntax e.g. `meson test regress:partition_join` that would make it
> consistent with the way TAP tests are run. But we need to make sure
> that the test later in the syntax would see the objects left behind by
> prior tests. E.g. `meson test regress:test_setup
> regress:partition_join` should see both tests passing. partition_join
> uses some tables created by test_setup, so those need to be run
> sequentially. Is that doable?

I think that makes sense, but it is not easily achievable right now.
The difference between TAP tests and regress/regress tests is that TAP
tests are registered individually, whereas regress/regress tests are
registered as one (with the --schedule option). This means we need to
register these tests one by one (instead of passing them with the
--schedule option) to the Meson build system in order to run them as
'meson test <test_group>:<test>'.

Additionally, the patch I shared earlier was only for regress/regress
tests. From what I understand from here [1], only regress/regress
tests support 'make check-tests', so the patch seems correct. I
experimented with how we can implement something similar for other
types of tests, including other regression, isolation, and ECPG tests.
The attached patch works for all types of tests but only for the Meson
builds. For example you can run:

$ meson test --suite setup
$ TESTS='check check_btree' meson test amcheck/regress
$ TESTS='ddl stream' meson test test_decoding/regress
$ TESTS='oldest_xmin skip_snapshot_restore' meson test test_decoding/isolation
$ TESTS='sql/prepareas compat_informix/dec_test' meson test ecpg/ecpg

What do you think about that behaviour? It is different from 'make
check-tests' but it looks useful to me.

[1] https://www.postgresql.org/message-id/1364.1717305911%40sss.pgh.pa.us

--
Regards,
Nazir Bilal Yavuz
Microsoft