Re: RFC: adding pytest as a supported test framework

On Thu, Jun 13, 2024 at 12:20 PM Robert Haas <robertmhaas(at)gmail(dot)com> wrote:
> I interpreted Jacob's original email as articulating a goal ("For the
> v18 cycle, I would like to try to get pytest [1] in as a supported
> test driver, in addition to the current offerings") rather than a
> plan.

That's the first part of it.

> There's no patch set yet and, as I understand it, no detailed
> plan for a patch set: that email seemed to focus on the question of
> desirability, rather than on outlining a plan of work, which I assume
> is still to come.

There was a four-step plan sketch at the end of that email, titled "A
Plan". That was not intended to be "the final detailed plan", because
I was soliciting feedback on the exact pieces people wanted to try to
implement first, and I am not the God Emperor of Pytest. But it was
definitely A Plan.

> Some things I'd like to see when a patch set does
> show up are:
>
> - good documentation for people who have no previous experience with
> Python and/or pytest e.g. here's how to set up your environment on
> Linux, Windows, macOS, *BSD so you can run the tests, here's how to
> run the tests, here's how it's different from the Perl framework we
> have now

+1

> - no external dependencies on PostgreSQL connectors. psql or libpq
> foreign function interface. the latter would be a cool increment of
> progress over the status quo.

If this is a -1 for psycopg, then I will cast my vote for ctypes/CFFI
and against psql.

> - at least as much in-tree support for writing tests as we have today
> with PostgreSQL::Test::whatever, but not necessarily a 1:1 reinvention
> of the stuff we have now, and documentation of those facilities that
> is as good or, ideally, better than what we have today.

I think this is way too much expectation for a v1 patch. If you were
committing this by yourself, would you agree to develop the entirety
of PostgreSQL::Test in a single commit, without the benefit of the
buildfarm checking you as you went, and other people trying to write
tests with it?

> - high overall code quality and level of maturity, not just something
> someone threw together for parity with the Perl system.

+1

> - enough tests written for or converted to the new system to give
> reviewers confidence that it's truly usable and fit for purpose.

This is that "know everything up front" tax that I think is not
reasonable for a test framework. If the thing you're trying to avoid
is the foot-in-the-door phenomenon, I would agree with you for a
Postgres feature. But these are tests; we don't ship them, we have
different rules for backporting them, they are developed in a very
different way.

> The important thing to me here (as it so often is) is to think like a
> maintainer. Imagine that immediately after the patches for this
> feature are committed, the developers who did the work all disappear
> from the community and are never heard from again. How much pain does
> that end us causing? The answer doesn't need to be zero; that is
> unrealistic. But it also shouldn't be "well, if that happens we're
> going to have to rip the feature out"

Can you elaborate on why that's not an okay outcome?

> or "well, a bunch of committers
> who didn't want to write tests in Python in the first place are now
> going to have to do a lot of work in Python to stabilize the work
> already committed."

Is it that? If the problem is that, we should address that. Because if
that is truly the fear, I cannot assuage that fear without showing you
something, and I cannot show you something you do not want to see, if
you don't want to write tests in Python in the first place.

--Jacob