Re: RFC: adding pytest as a supported test framework

On Sat, 15 Jun 2024 at 16:45, Andrew Dunstan <andrew(at)dunslane(dot)net> wrote:
> I see the fact that we stash the output in a file as a feature. Without
> it, capturing failure information in the buildfarm client would be more
> difficult, especially if there are multiple failures. So this is
> actually something I think we would need for any alternative framework.

I indeed heard that the current behaviour was somehow useful to the
buildfarm client.

> Maybe we need an environment setting that would output the
> regress_log_00whatever file to stderr on failure. That should be pretty
> easy to arrange in the END handler for PostgreSQL::Test::Utils.

That sounds awesome! But I'm wondering: do we really need a setting to
enable/disable that? Can't we always output it to stderr on failure?
If we output the log both to stderr and as a file, would that be fine
for the build farm? If not, a setting should work. (but I'd prefer the
default for that setting to be on in that case, it seems much easier
to turn it off in the buildfarm client, instead of asking every
developer to turn the feature on)

> > 4. Pytest has autodiscovery of test files and functions, so we
> > probably wouldn't have to specify all of the exact test files anymore
> > in the meson.build files.
>
>
> I find this comment a bit ironic. We don't need to do that with the
> Makefiles, and the requirement to do so was promoted as a meson feature
> rather than a limitation, ISTR.

Now, I'm very curious why that would be considered a feature. I
certainly have had many cases where I forgot to add the test file to
the meson.build file.

> > Regarding 2, there are ~150 checks that are using a suboptimal way of
> > testing for a comparison. Mostly a lot that could use "like(..., ...)"
> > instead of "ok(... ~= ...)"
> > ❯ grep '\bok(.*=' **.pl | wc -l
> > 151
>
>
> Well, let's fix those. I would be tempted to use cmp_ok() for just about
> all of them.

Sounds great to me.

> But the fact that Test::More has a handful of test primitives rather
> than just one strikes me as a relatively minor complaint.

It is indeed a minor paper cut, but paper-cuts add up.

Honestly, my primary *objective* complaint about our current test
suite, is that when a test fails, it's very often impossible for me to
understand why the test failed, by only looking at the output of
"meson test". I think logging the postgres log to stderr for Perl, as
you proposed, would significantly improve that situation. I think the
only thing that we cannot get from Perl Test::More that we can from
pytest, is the fancy recursive introspection of the expression that
pytest shows on error.

Apart from that my major *subjective* complaint is that I very much
dislike writing Perl code. I'm slow at writing it and I don't (want
to) improve at it because I don't have reasons to use it except for
Postgres tests. So currently I'm not really incentivised to write more
tests than the bare minimum, help improve the current test tooling, or
add new testing frameworks for things we currently cannot test.
Afaict, there's a significant part of our current community who feel
the same way (and I'm pretty sure every sub-30 year old person who
newly joins the community would feel the exact same way too).

As a project I think we would like to have more tests, and to have
more custom tooling to test things that we currently cannot (e.g.
oauth or manually messing with the wire-protocol). I think the only
way to achieve that is by encouraging more people to work on these
things. I very much appreciate that you and others are improving our
Perl tooling, because that makes our current tests easier to work
with. But I don't think it significantly increases the willingness to
write tests or test-tooling for people that don't want to write Perl
in the first place.

So I think the only way to get more people involved in contributing
tests and test-tooling is by allowing testing in another language than
Perl (but also still allow writing tests in Perl). Even if that means
that we have two partially-overlapping test frameworks, that are both
easy to use for different things. In my view that's even a positive
thing, because that means we are able to test more with two languages
than we would be able to with either one (and it's thus useful to have
both).

And I agree with Robbert that Python seems like the best choice for
this other language, given its current popularity level. But as I said
before, I'm open to other languages as well.