Re: RFC: adding pytest as a supported test framework

Hi,

On 2024-06-15 10:45:16 -0400, Andrew Dunstan wrote:
> > 4. Pytest has autodiscovery of test files and functions, so we
> > probably wouldn't have to specify all of the exact test files anymore
> > in the meson.build files.
>
>
> I find this comment a bit ironic. We don't need to do that with the
> Makefiles, and the requirement to do so was promoted as a meson feature
> rather than a limitation, ISTR.

The reason its good to have the list of tests somewhere explicit is that we
have different types of test. With make, there is a single target for all tap
tests. If you want to run tests concurrently, make can only schedule the tap
tests at the granularity of a directory. If you want concurrency below that,
you need to use concurrency on the prove level. But that means that you have
extremely varying concurrency, depending on whether make runs targets that
have no internal concurrency or make runs e.g. the recovery tap tests.

I don't think we should rely on global test discovery via pytest. That'll lead
to uncontrollable concurrency again, which means much longer test times. We'll
always have different types of tests, just scheduling running them via
"top-level" tools for different test types just won't work well. That's not
true for many projects where tests have vastly lower resource usage.

Greetings,

Andres Freund