Re: [COMMITTERS] pgsql: Add TAP tests for pg_dump

Peter,

* Peter Eisentraut (peter(dot)eisentraut(at)2ndquadrant(dot)com) wrote:
> On 5/7/16 9:36 AM, Stephen Frost wrote:
> >Honestly, over the next couple of months between feature-freeze and
> >release, I'd like to add even more tests, and not just to pg_dump but
> >also to other commands that don't have very good testing today (psql, in
> >particular, but pg_dumpall needs more also, and there's more to do with
> >pg_dump too).
>
> If we're going to do that, there will be no stopping it. I also
> have a bunch of code in this area lined up, but I was hoping to
> submit it to the commit fest process at the right time and order to
> get feedback and testing. If we're going to allow new tests being
> developed right up until release, I can just stop doing release
> preparation work right now and get back to coding and reviewing.

I do think that now is a good time for people to be reviewing what's
been committed, which includes writing tests (either automated ones,
which can be included in our test suite, or one-off's for testing
specific things but which don't make sense to include).

That doesn't mean we should be codeing or reviewing new features for
commit.

As for release prep, I certainly applaud everyone involved in that
effort as we do have the beta release and back-branch releases coming
up.

Regarding when we should stop adding new tests, I can't agree with the
notion that they should be treated as new features. New tests may break
the buildfarm, but they do not impact the stability of committed code,
nor do they introduce new bugs into the code which has been committed
(if anything, they expose committed bugs, as the set of tests we're
discussing did, which should then be fixed).

> Ultimately, tests are code and should be treated as such.

Perhaps in some manners this is accurate, but I'd view our test suite as
an independent code base from PG. Bugs in the test suite might cause
false test failures or other issues on the buildfarm or during
packaging, but bugs or issues in the test suite won't cause data loss,
data corruption, or generally impact running operations of our users.

I can see some sense in holding off on adding more tests once we hit RC,
as we want anything done with RC to be essentially identical to the
release, unless there is a serious issue, but holding off on adding new
tests which could expose issues in committed code for the months during
beta doesn't seem sensible.

As such, I'd certainly encourage you to propose new tests, even now, but
not changes to the PG code, except for bug fixes, or changes agreed to
by the RMT. How you prioritise that with the release preparation work
is up to you, of course, though if I were in your shoes, I'd take care
of the release prep first, as we're quite close to doing a set of
releases. I'm happy to try and help with that effort myself, though
I've not been involved very much in release prep and am not entirely
sure what I can do to help.

Thanks!

Stephen