Re: RFC: adding pytest as a supported test framework

Hi,

On 2024-06-14 11:49:29 -0400, Tom Lane wrote:
> I also wonder about integration of python-based testing with what
> we already have. A significant part of what you called the meson
> work had to do with persuading pg_regress, isolationtester, etc
> to output test results in the common format established by TAP.

FWIW, meson's testrunner doesn't require TAP, the lowest common denominator is
just an exit code. However, for things that run many "sub" tests, it's a lot
nicer if the failures can be reported more granularly than just "the entire
testsuite failed".

Meson currently supports:

exitcode: the executable's exit code is used by the test harness to record the outcome of the test).

tap: Test Anything Protocol.

gtest (since 0.55.0): for Google Tests.

rust (since 0.56.0): for native rust tests

> Am I right in guessing that pytest will have nothing to do with that?

Looks like there's a plugin for pytest to support tap as output:
https://pypi.org/project/pytest-tap/

However, it's not available as a debian package. I know that some folks just
advocate installing dependencies via venv, but I personally don't think
that'll fly. For one, it'll basically prevent tests being run by packagers.

> Can we even manage to dump perl and python test scripts into the same
> subdirectory and sort things out automatically?

That shouldn't be much of a problem.

> I'm definitely going to be -1 for a python testing feature that cannot
> integrate with what we have because it demands its own control and
> result-collection infrastructure.

Dito.

Greetings,

Andres Freund