Re: On Judging the Value of Tests

Craig Ringer <craig(at)2ndquadrant(dot)com> writes:
> On 22 November 2017 at 08:43, Jankirk.Vincent., Jamison <
> k(dot)jamison(at)jp(dot)fujitsu(dot)com> wrote:
>> 1. How do you judge when a test suite is acceptable to be added to
>> Postgres OSS source code? (How do you judge the value of a test suite?)

> Make your argument for it, and see if others agree. There's no formal
> process.

Yeah. Also realize that we've been accreting test cases for ~20 years,
and there's never been any master plan about what to test. So there's a
lot of individual test cases of varying quality/value, and you shouldn't
necessarily take any one existing test as gospel. I don't hold that out
as a great example of software engineering, but it's the truth.

> Inputs into the decision making process include:

> * How much coverage of previously untested functionality it adds
> * How much code coverage it adds
> * How long the test takes to run, especially considering the slow buildfarm
> boxes and development turnaround time
> * Whether the test fits into one of the existing suites we run routinely,
> or requires separate steps
> * How much work will be required to maintain the test

Also, portability/repeatability. If it doesn't produce the same answers,
with very high probability, across all the platforms we support, it isn't
going to last long.

>> 3. Is there a standard way of writing tests on the source code that
>> we should follow, like when should test be written in TAP/SQL/C formats and
>> how long should it be?

> In general, prefer pg_regress if you can use it. Isolation tests for
> concurrency issues. TAP tests if you can't write it with pg_regress or
> isolation tester. Test modules only if you really must.

Right. This ties in closely to the incremental runtime needed for one
additional test case. A new query or two in an existing SQL regression
script adds little overhead. A new TAP test adds quite a bit.

>> 4. In the src/test/examples directory (which are all libpq tests),
>> why is the “examples” directory not included when building postgres?

I've always taken those to be more documentation, or sample code, than
test cases. I'm not sure why they're under src/test/ at all.

regards, tom lane