Re: RFC: adding pytest as a supported test framework

On 2024-06-14 Fr 18:11, Jelte Fennema-Nio wrote:
> On Fri, 14 Jun 2024 at 17:49, Tom Lane<tgl(at)sss(dot)pgh(dot)pa(dot)us> wrote:
>> But what I'd really like to see is some comparison of the
>> language-provided testing facilities that we're proposing
>> to depend on. Why is pytest (or whatever) better than Test::More?
> Some advantages of pytest over Test::More:
>
> 1. It's much easier to debug failing tests using the output that
> pytest gives. A good example of this is on pytest its homepage[1]
> (i.e. it shows the value given to the call to inc in the error)
> 2. No need to remember a specific comparison DSL
> (is/isnt/is_deeply/like/ok/cmp_ok/isa_ok), just put assert in front of
> a boolean expression and your output is great (if you want to add a
> message too for clarity you can use: assert a == b, "the world is
> ending")
> 3. Very easy to postgres log files on stderr/stdout when a test fails.
> This might be possible/easy with Perl too, but we currently don't do
> that. So right now for many failures you're forced to traverse the
> build/testrun/... directory tree to find the logs.

I see the fact that we stash the output in a file as a feature. Without
it, capturing failure information in the buildfarm client would be more
difficult, especially if there are multiple failures. So this is
actually something I think we would need for any alternative framework.

Maybe we need an environment setting that would output the
regress_log_00whatever file to stderr on failure.  That should be pretty
easy to arrange in the END handler for PostgreSQL::Test::Utils.

> 4. Pytest has autodiscovery of test files and functions, so we
> probably wouldn't have to specify all of the exact test files anymore
> in the meson.build files.

I find this comment a bit ironic. We don't need to do that with the
Makefiles, and the requirement to do so was promoted as a meson feature
rather than a limitation, ISTR.

> Regarding 2, there are ~150 checks that are using a suboptimal way of
> testing for a comparison. Mostly a lot that could use "like(..., ...)"
> instead of "ok(... ~= ...)"
> ❯ grep '\bok(.*=' **.pl | wc -l
> 151

Well, let's fix those. I would be tempted to use cmp_ok() for just about
all of them.

But the fact that Test::More has a handful of test primitives rather
than just one strikes me as a relatively minor complaint.

cheers

andrew

--

Andrew Dunstan
EDB: https://www.enterprisedb.com