Re: On Judging the Value of Tests

On Wed, Nov 22, 2017 at 10:31 AM, Craig Ringer <craig(at)2ndquadrant(dot)com> wrote:
> On 22 November 2017 at 08:43, Jankirk.Vincent., Jamison
> <k(dot)jamison(at)jp(dot)fujitsu(dot)com> wrote:
>> I am entirely very new to PG development.
>>
>> Currently, I’m studying the test suites, and I feel the more experienced
>> PG devs here could provide some insights on testing because I could not find
>> concrete answers in the Postgres documentations.
>>
>> 1. How do you judge when a test suite is acceptable to be added to
>> Postgres OSS source code? (How do you judge the value of a test suite?)
>
> Make your argument for it, and see if others agree. There's no formal
> process.
>
> Inputs into the decision making process include:
>
> * How much coverage of previously untested functionality it adds
> * How much code coverage it adds
> * How long the test takes to run, especially considering the slow buildfarm
> boxes and development turnaround time
> * Whether the test fits into one of the existing suites we run routinely, or
> requires separate steps
> * How much work will be required to maintain the test

Even with those factors in mind there is a case-by-case variability.
Sometimes it is not worth initializing a new instance with initdb for
a small test. Parallelization of tests help on large machines by on
buildfarm members the size of a RPI things get time-consuming.
Priority is usually given into things that do not consume much time,
so as any developer can run the full set of tests quickly on a laptop.
I have for example a small alias to run the full set of basic tests
(make check-world -j 4 PROVE_FLAGS="-j 4") and this finishes within
two minutes if I recall correctly, TAP tests included. So runtime is
really an important matter.

>> 3. Is there a standard way of writing tests on the source code that
>> we should follow, like when should test be written in TAP/SQL/C formats and
>> how long should it be?
>>
>> ① I know that TAP test is for client program tests, SQL for
>> regression tests with sql queries, but there are instances also where tests
>> are also written in C like isolation tests, etc. How do we best judge which
>> language is preferred to use when writing tests for Postgres components? How
>> long should a test be when proposing them to hackers page?
>
> In general, prefer pg_regress if you can use it. Isolation tests for
> concurrency issues. TAP tests if you can't write it with pg_regress or
> isolation tester. Test modules only if you really must.

This really depends on what you want to achieve, so this comes back to
a case-by-case. Regression tests are fine for basic tests, where one
session is enough to prove your feature. Isolation tests are helpful
when testing concurrent behaviors across multple sessions. TAP tests
are useful when more than one instance is needed or when you need to
test cluster-wide configurations or client binaries. Having tests in
patches proposed definitely helps people in understanding more how
something works. Sometimes things get discarded but they can really
help in making a point. Lately, while working on channel binding
support for SCRAM, TAP tests have for example proved to help:
https://git.postgresql.org/gitweb/?p=postgresql.git;a=commitdiff;h=9288d62bb4b6f302bf13bb2fed3783b61385f315#patch13.
This needs a SSL setup with a specific configuration for pg_hba.conf,
and proved with just a lookup at the tests in the patch that the
feature proposed was able to achieve what it aimed at. So it proved a
point.

>> 4. In the src/test/examples directory (which are all libpq tests),
>> why is the “examples” directory not included when building postgres? (Why
>> weren't these libpq tests added to src/interface/libpq/test or in regression
>> test suite instead?) In short, how to know where (in which file/directory in
>> source code) to put a test?
>
> Dunno, sorry.

Those are mainly present as example programs.
--
Michael