Re: Cost of src/test/recovery and .../subscription tests

On Wed, Apr 19, 2017 at 1:28 PM, Tom Lane <tgl(at)sss(dot)pgh(dot)pa(dot)us> wrote:
> So I updated longfin to the new release of the buildfarm client,
> and was quite dismayed by the fact that its cycle time went
> from 16 minutes to 24. Some of that might be random effects like
> the state of the kernel disk caches, but a large chunk of it
> --- over 5 minutes --- evidently is from src/test/recovery/,
> which the buildfarm script didn't run before and now does.
>
> I am going to say flat out that that's unacceptably long for
> a test script that will be run dozens of times a day by the
> buildfarm. There isn't any other test script that takes more
> than circa 90 seconds on that machine, and I don't think this
> one should either.

I think that's bunk. If there are tests that are part of those test
suites that are taking a long time to run and not providing meaningful
coverage, then that's something that can be improved. However, I
reject the argument that a test running for a long time is in itself
bad. I'd rather have tests that run for a long time (and thus get run
less often) than have no tests.

Mind you, I'm not entirely sanguine about the large increase in time
that it takes to run make check-world on my local machine vs. a few
years ago. It's annoying, and optimizing it would be good. I think
the TAP test framework encourages people to add possibly-lengthy tests
pretty freely, because you can stick the test inside of a loop and run
it in many slightly different configurations, so people do, maybe
without giving enough thought to the value of those tests. But
outlawing long-running tests is not the right solution to that
problem.

--
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company