Re: RFC: adding pytest as a supported test framework

On Thu, Jun 13, 2024 at 2:52 PM Jelte Fennema-Nio <postgres(at)jeltef(dot)nl> wrote:
> I understand and agree with your final stated goal of not ending up in
> another big mess. It's also clear to me that you don't think the
> current proposal achieves that goal. So I assume you have some
> additional ideas for the proposal to help achieve that goal and/or
> some specific worries that you'd like to get addressed better in the
> proposal. But currently it's not really clear to me what either of
> those are. Could you clarify?

Hmm, I don't know that I have what you're hoping I have, or at least
not any more than what I've said already.

I interpreted Jacob's original email as articulating a goal ("For the
v18 cycle, I would like to try to get pytest [1] in as a supported
test driver, in addition to the current offerings") rather than a
plan. There's no patch set yet and, as I understand it, no detailed
plan for a patch set: that email seemed to focus on the question of
desirability, rather than on outlining a plan of work, which I assume
is still to come. Some things I'd like to see when a patch set does
show up are:

- good documentation for people who have no previous experience with
Python and/or pytest e.g. here's how to set up your environment on
Linux, Windows, macOS, *BSD so you can run the tests, here's how to
run the tests, here's how it's different from the Perl framework we
have now

- no external dependencies on PostgreSQL connectors. psql or libpq
foreign function interface. the latter would be a cool increment of
progress over the status quo.

- at least as much in-tree support for writing tests as we have today
with PostgreSQL::Test::whatever, but not necessarily a 1:1 reinvention
of the stuff we have now, and documentation of those facilities that
is as good or, ideally, better than what we have today.

- high overall code quality and level of maturity, not just something
someone threw together for parity with the Perl system.

- enough tests written for or converted to the new system to give
reviewers confidence that it's truly usable and fit for purpose.

The important thing to me here (as it so often is) is to think like a
maintainer. Imagine that immediately after the patches for this
feature are committed, the developers who did the work all disappear
from the community and are never heard from again. How much pain does
that end us causing? The answer doesn't need to be zero; that is
unrealistic. But it also shouldn't be "well, if that happens we're
going to have to rip the feature out" or "well, a bunch of committers
who didn't want to write tests in Python in the first place are now
going to have to do a lot of work in Python to stabilize the work
already committed."

--
Robert Haas
EDB: http://www.enterprisedb.com