Re: New regression test time

On 2 July 2013 18:43, Noah Misch <noah(at)leadboat(dot)com> wrote:

> On Tue, Jul 02, 2013 at 10:17:08AM -0400, Robert Haas wrote:
> > So I think the first question we need to answer is: Should we just
> > reject Robins' patches en masse? If we do that, then the rest of this
> > is moot. If we don't do that, then the second question is whether we
> > should try to introduce a new schedule, and if so, whether we should
> > split out that new schedule before or after committing these patches
> > as they stand.
>
> It's sad to simply reject meaningful automated tests on the basis of doubt
> that they're important enough to belong in every human-in-the-loop test
> run.
> I share the broader vision for automated testing represented by these
> patches.

+1 We should be encouraging people to submit automated tests.

I share the annoyance of increasing the length of the automated test runs,
and have watched them get longer and longer over time. But I run the tests
because I want to see whether I broke anything and I stopped sitting and
watching them run long ago.

Automated testing is about x10-100 faster than manual testing, so I see new
tests as saving me time not wasting it.

> > Here are my opinions, for what they are worth. First, I think that
> > rejecting these new tests is a bad idea, although I've looked them
> > over a bit and I think there might be individual things we might want
> > to take out. Second, I think that creating a new schedule is likely
> > to cost developers more time than it saves them. Sure, you'll be able
> > to run the tests slightly faster, but when you commit something, break
> > the buildfarm (which does run those tests), and then have to go back
> > and fix the tests (or your patch), you'll waste more time doing that
> > than you saved by avoiding those few extra seconds of runtime. Third,
> > I think if we do want to create a new schedule, it makes more sense to
> > commit the tests first and then split out the new schedule according
> > to some criteria that we devise. There should be a principled reason
> > for putting tests in one schedule or the other; "all the tests that
> > Robins Tharakan wrote" is not a good filter criteria.
>
> +1 for that plan. I don't know whether placing certain tests outside the
> main
> test sequence would indeed cost more time than it saves. I definitely
> agree
> that if these new tests should appear elsewhere, some of our existing tests
> should also move there.
>

Let's have a new schedule called minute-check with the objective to run the
common tests in 60 secs.

We can continue to expand the normal schedules from here.

Anybody that wants short tests can run that, everyone else can run the full
test suite.

We should be encouraging people to run the full test suite, not the fast
one.

--
Simon Riggs http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Training & Services