Re: new tests post-feature freeze (was pgsql: Add TAP tests for pg_dump)

On Sat, May 7, 2016 at 5:44 PM, Stephen Frost <sfrost(at)snowman(dot)net> wrote:
> * Peter Eisentraut (peter(dot)eisentraut(at)2ndquadrant(dot)com) wrote:
>> On 5/7/16 9:36 AM, Stephen Frost wrote:
>> >Honestly, over the next couple of months between feature-freeze and
>> >release, I'd like to add even more tests, and not just to pg_dump but
>> >also to other commands that don't have very good testing today (psql, in
>> >particular, but pg_dumpall needs more also, and there's more to do with
>> >pg_dump too).
>>
>> If we're going to do that, there will be no stopping it. I also
>> have a bunch of code in this area lined up, but I was hoping to
>> submit it to the commit fest process at the right time and order to
>> get feedback and testing. If we're going to allow new tests being
>> developed right up until release, I can just stop doing release
>> preparation work right now and get back to coding and reviewing.
>
> I do think that now is a good time for people to be reviewing what's
> been committed, which includes writing tests (either automated ones,
> which can be included in our test suite, or one-off's for testing
> specific things but which don't make sense to include).
>
> That doesn't mean we should be codeing or reviewing new features for
> commit.
>
> As for release prep, I certainly applaud everyone involved in that
> effort as we do have the beta release and back-branch releases coming
> up.
>
> Regarding when we should stop adding new tests, I can't agree with the
> notion that they should be treated as new features. New tests may break
> the buildfarm, but they do not impact the stability of committed code,
> nor do they introduce new bugs into the code which has been committed
> (if anything, they expose committed bugs, as the set of tests we're
> discussing did, which should then be fixed).
>
>> Ultimately, tests are code and should be treated as such.
>
> Perhaps in some manners this is accurate, but I'd view our test suite as
> an independent code base from PG. Bugs in the test suite might cause
> false test failures or other issues on the buildfarm or during
> packaging, but bugs or issues in the test suite won't cause data loss,
> data corruption, or generally impact running operations of our users.
>
> I can see some sense in holding off on adding more tests once we hit RC,
> as we want anything done with RC to be essentially identical to the
> release, unless there is a serious issue, but holding off on adding new
> tests which could expose issues in committed code for the months during
> beta doesn't seem sensible.
>
> As such, I'd certainly encourage you to propose new tests, even now, but
> not changes to the PG code, except for bug fixes, or changes agreed to
> by the RMT. How you prioritise that with the release preparation work
> is up to you, of course, though if I were in your shoes, I'd take care
> of the release prep first, as we're quite close to doing a set of
> releases. I'm happy to try and help with that effort myself, though
> I've not been involved very much in release prep and am not entirely
> sure what I can do to help.

In the end, the question of the degree to which tests constitute
features is one that needs to be made by the whole community, not
individual developers. I think you both raise good points. On the one
hand, developing new test frameworks could distract from other release
preparation tasks, as Peter rightly points out. On the other hand, it
could also make the release more reliable, as Stephen points out. I
believe that there is a clear consensus that at least some new tests
are welcome even post-feature freeze. However, I also believe that we
could get carried away with that, and end up having it become a
distraction from the task of getting the release out the door.

Also, if we say that new tests are not features, that would mean that
they could be back-patched even after the release is out the door, and
generally I'm not in favor of that policy, except when we're adding a
test to a back-branch that is closely tied to a bug we are fixing in
that branch. I do not want to see the pg_dump test suite back-patched
to all active branches, for example, even though I approve of having
test coverage for pg_dump. I am confident that minimizing churn in
the back-branches is a more important goal than ensuring test coverage
for those branches, and that we will regret it if we reverse those
priorities.

My suggestion is that, from this point forward, we add new tests to
9.6 only if they are closely related to a bug that is getting fixed or
a feature that is new in 9.6. I think that's a reasonable compromise,
but what do others think?

--
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company