Re: [COMMITTERS] pgsql: Add TAP tests for pg_dump

On 5/9/16, Stephen Frost <sfrost(at)snowman(dot)net> wrote:
>> And what if the tests are buggy? New test suites should go through a
>> CF first I think for proper review. And this is clearly one.
>
> They still won't result in data loss, corruption, or other direct impact
> on our users, even if they are buggy. They also won't destablize the
> code base or introduce new bugs into the code.

Yes, but they could make things worse long term. For example if
introduce intermittent failures or if they're hard to understand or if
they test internals too deeply and don't let those internals evolve
because the tests break. All of them reasons why it's good that
they're reviewed.

> There is pretty clearly a lack of consensus on the question about adding
> new tests.

> What *should* we be doing right now? Reviewing code and testing the
> system is what I had thought but perhaps I'm wrong. To the extent that
> we're testing the system, isn't it better to write repeatable tests,
> particularly ones which add code coverage, than one-off tests that only
> check that the current snapshot of the code is operating correctly?

I also think that testing *and* keeping those tests for the future is
more valuable than just testing. But committers can just push their
tests while non committers need to wait for the next CF to get their
new tests committed. And committers pushing tests means they bypass
the review process which, as far as I know, doesn't happen for regular
patches: committers usually only directly commit small fixes not lots
of new (test) code.

So, crazy idea: what about test only CommitFests in between now and
the release? The rules would be: only new tests and existing test
improvements are allowed. For the rest, the CF would be the same as
usual: have an end date, committers agree to go through each patch and
commit or return it with feedback etc. Stephen would have added his
tests to the upcoming June test CF, Michael would have reviewed them
and maybe add his own and so on.

This does create work for committers (go through the submitted
patches) but acts as an incentive for test writers. Want your test
code committed? Well, there will be a CF very soon where it will get
committer attention so better write that test. It also gives a clear
date after which test churn also stops in order to not destabilize the
buildfarm right before a release.