Re: Running tests under valgrind is getting slower at an alarming pace

Andres Freund <andres(at)anarazel(dot)de> writes:
> After a recent migration of the skink and a few other animals (sorry for the
> false reports on BF, I forgot to adjust a path), I looked at the time it takes
> to complete a valgrind run:

> 9.6: Consumed 4h 53min 18.518s CPU time
> 10: Consumed 5h 32min 50.839s CPU time
> 11: Consumed 7h 7min 17.455s CPU time
> 14: still going at 11h 51min 57.951s
> HEAD: 14h 32min 29.571s CPU time

I have observed similar slowdowns across versions on just-plain-slow
animals, too. Awhile ago (last December, I think), I tried enabling
--enable-tap-tests across the board on prairiedog, and observed
these buildfarm cycle times:

9.5 01:50:24
9.6 02:06:32
10 02:26:34
11 02:54:44
12 03:41:11
13 04:46:31
HEAD 04:49:04

I went back to not running TAP tests in the back branches :-(

prairiedog's latest HEAD run consumed 5:30, so it's gotten way
worse since December.

In the same comparison, my other animal longfin had gone from 14 to 18
minutes (and it's now up to 22 minutes). It's not clear to me whether
greater available parallelism (12 CPUs vs 1) is alone enough to
explain why the more modern machine isn't suffering so badly. As you
say, the TAP tests are not well parallelized at all, so that doesn't
seem to fit the facts.

In any case, it seems like we do need to be paying more attention to
how long it takes to do the TAP tests. We could try to shave more
cycles off the overhead, and we should think a little harder about
the long-term value of every test case we add.

regards, tom lane