RE: On Judging the Value of Tests

>>>> 1. How do you judge when a test suite is acceptable to be added to
>>>> Postgres OSS source code? (How do you judge the value of a test
>>>> suite?)
>>>
>>> Make your argument for it, and see if others agree. There's no formal
>>> process.
>
> Yeah. Also realize that we've been accreting test cases for ~20 years,
> and there's never been any master plan about what to test. So there's a
> lot of individual test cases of varying quality/value, and you shouldn't
> necessarily take any one existing test as gospel. I don't hold that out
> as a great example of software engineering, but it's the truth.
>>>
>>> Inputs into the decision making process include:
>>> * How much coverage of previously untested functionality it adds
>>> * How much code coverage it adds
>>> * How long the test takes to run, especially considering the slow
>>> buildfarm boxes and development turnaround time
>>> * Whether the test fits into one of the existing suites we run
>>> routinely, or requires separate steps
>>> * How much work will be required to maintain the test
>>
>> Even with those factors in mind there is a case-by-case variability.
>> Sometimes it is not worth initializing a new instance with initdb for
>> a small test. Parallelization of tests help on large machines by on
>> buildfarm members the size of a RPI things get time-consuming.
>> Priority is usually given into things that do not consume much time,
>> so as any developer can run the full set of tests quickly on a laptop.
>> I have for example a small alias to run the full set of basic tests
>> (make check-world -j 4 PROVE_FLAGS="-j 4") and this finishes within
>> two minutes if I recall correctly, TAP tests included. So runtime is
>> really an important matter.
>
> Also, portability/repeatability. If it doesn't produce the same answers,
> with very high probability, across all the platforms we support, it isn't
> going to last long.

Thank you everyone for your valuable input.
I will take note of all your advice. I think it's because I've noticed that a lot of functions and lines are still not covered by tests (https://coverage.postgresql.org/), so I feel like I want to work on tests first while I'm still learning the PG internals and hacking.

>>>> 3. Is there a standard way of writing tests on the source code that
>>>> we should follow, like when should test be written in TAP/SQL/C
>>>> formats and how long should it be?
>>>
>>> In general, prefer pg_regress if you can use it. Isolation tests for
>>> concurrency issues. TAP tests if you can't write it with pg_regress
>>> or isolation tester. Test modules only if you really must.
>
> Right. This ties in closely to the incremental runtime needed for one
> additional test case. A new query or two in an existing SQL regression
> script adds little overhead. A new TAP test adds quite a bit.

I understand that tests should not take that long when running.
For example, how about when proposing tests that just test
parameter settings of a function?
Let's say, for example, a libpq main function where in different test cases
the params are null, empty, invalid, or valid.
Are these test types common or recommended?

>>>> 4. In the src/test/examples directory (which are all libpq tests),
>>>> why is the “examples” directory not included when building postgres?
>>>
>>> Dunno, sorry.
>>
>> Those are mainly present as example programs.
>
> I've always taken those to be more documentation, or sample code, than
> test cases. I'm not sure why they're under src/test/ at all.

Since these example codes are only documentations, so should these be removed in the source code instead? Or should these examples code be revised and be moved to a test directory in libpq?

Thanks again!

Regards,
Kirk Jamison