Re: Contributing test cases to improve coverage

On Thu, Jun 13, 2024 at 1:22 AM J F <jonathanfoo0523(at)gmail(dot)com> wrote:

> > I am quite confused about what is the point of this. You have not
> > found any actual bug, nor have you demonstrated that this test case
> > could discover a likely future bug that wouldn't be detected another
> > way. Moreover, it seems like the process would lead to some very
> > large number of equally marginal test cases. We aren't likely to
> > accept such a patch, because we are concerned about keeping down the
> > runtime of the test suite.
> >
> > regards, tom lane
>
>
> The point of this project is to improve the coverage of PostgreSQL’s
> preexisting test suite. Writing a test suite to achieve close to 100%
> coverage is challenging, but I have proposed a workflow to automate this
> process.
>

We monitor code coverage. But this is doing more than that. It will find
out the places in code, which if changed, will cause bugs. That seems
useful to avoid refactoring mistakes, esp. when people rely on regression
tests to tell whether their code changes are sane. But in PostgreSQL, we
rely heavily on reviewers and committers to do that instead of tests.
Still, the tests produced by this tool will help catch bugs that human eyes
can not. As Tom said, managing that battery of tests may not be worth it.
Basically, I am flip-flopping on the usefulness of this effort.

>
> I assert that no test case in the regression test suite currently covers
> the comparator in the expression rte->rtekind == RTE_SUBQUERY. I propose
> adding a new test case that addresses exactly this. In the future, if
> someone accidentally modifies the operator to become >=, it will trigger
> incorrect behavior when certain queries are executed. This test case will
> catch that issue.
>

Usually PostgreSQL developers know that rtekind is an enum so they are very
very unlikely to use anything other than == and !=. Such a change will be
caught by a reviewer. I think many of the tests that this tool will produce
will fall in this category.

>
> I get that the test cases in /regress are likely reserved for actual bugs
> found and are designed to run quickly. Would it be a good idea to have a
> separate, more rigorous test suite that runs longer but provides better
> code coverage?
>
> There are practical difficulties like maintaining the expected outputs for
such a large battery of tests. But maybe some external project could.

BTW, have you considered perl tests, isolation tests etc. Tests in regress/
do not cover many subsystems e.g. replication.
--
Best Wishes,
Ashutosh Bapat