Re: Test code is worth the space

* Robert Haas (robertmhaas(at)gmail(dot)com) wrote:
> On Wed, Aug 12, 2015 at 2:04 PM, Peter Geoghegan <pg(at)heroku(dot)com> wrote:
> > This resistance to adding tests seems quite short sighted to me,
> > especially when the concern is about queries that will each typically
> > take less than 1ms to execute. Like Noah, I think that it would be
> > very helpful to simply be more inclusive of additional tests that
> > don't increase test coverage by as much as each query in a minimal
> > subset. I am not at all convinced by arguments about the cost of
> > maintaining tests when a simple behavioral change occurs.
>
> I've removed tests from patches that in my opinion were unlikely to
> fail either (a) for any reason or (b) for any reason other than an
> intentional change, and I think that's a reasonable thing to do.
> However, I still think it's a good idea, and useful, to try to expand
> the code coverage we get from 'make check'. However, the bigger
> issue, IMHO, is the stuff that can't be tested via pg_regress, e.g.
> because it needs hooks, like what Alvaro is talking about, or because
> it needs a custom testing framework. Recovery, for example, really
> needs a lot more testing, as we talked about at PGCon. If we just
> expand what 'make check' covers and don't deal with those kinds of
> things, we will be improving our test coverage but maybe not all that
> much.

Agreed, and this is something which I said we'd work to help with and
which we have some things to show now, for those interested.

The regression tests included in pgBackRest (available here:
https://github.com/pgmasters/backrest) go through a number of different
recovery tests. There's vagrant configs for a few different VMs too
(CentOS 6, CentOS 7, Ubuntu 12.04 and Ubuntu 14.04) which is what we've
been testing.

We're working to continue expanding those tests and will also be adding
tests for replication and promotion in the future. Eventually, we plan
to write a buildfarm module for pgBackRest, to allow it to be run in the
same manner as the regular buildfarm animals with the results posted.

David, feel free to correct me if I'm misconstrued anything above.

Thanks!

Stephen