Re: Test code is worth the space

On Thu, Aug 13, 2015 at 1:31 AM, Robert Haas <robertmhaas(at)gmail(dot)com> wrote:

> On Wed, Aug 12, 2015 at 7:20 PM, Tom Lane <tgl(at)sss(dot)pgh(dot)pa(dot)us> wrote:
> > FWIW, I've objected in the past to tests that would significantly
> > increase the runtime of "make check", unless I thought they were
> > especially valuable (which enumerating every minor behavior of a
> > feature patch generally isn't IMO). I still think that that's an
> > important consideration: every second you add to "make check" is
> > multiplied many times over when you consider how many developers
> > run that how many times a day.
> >
> > We've talked about having some sort of second rank of tests that
> > people wouldn't necessarily run before committing, and that would
> > be allowed to eat more time than the core regression tests would.
> > I think that might be a valuable direction to pursue if people start
> > submitting very bulky tests.
>
> Maybe. Adding a whole new test suite is significantly more
> administratively complex, because the BF client has to get updated to
> run it. And if expected outputs in that test suite change very often
> at all, then committers will have to run it before committing anyway.
>

We could always do that the other way - meaning put everything in "make
check", and invent a "make quickcheck" for developers with the smaller
subset.

> The value of a core regression suite that takes less time to run has
> to be weighed against the possibility that a better core regression
> suite might cause us to find more bugs before committing. That could
> easily be worth the price in runtime.
>

Or have a quickcheck you run "all the time" and then run the bigger one
once before committing perhaps?

--
Magnus Hagander
Me: http://www.hagander.net/
Work: http://www.redpill-linpro.com/