distributed performance testing

A little while ago I rather rashly opined that we could extend the
buildfarm to do (optional) performance testing. After thinking about it
for some time I now think that might not be such a good idea. We can
certainly leverage a lot of the technology used and experience gained in
building the buildfarm to create a distributed performance testing
system, and we know that many among the buildfarm members would be
willing to join such an effort, but I now think it probably needs to be
a separate beast.

Buildfarm tests correctness on various code branches, for a fairly
static config set per member, triggered by code changes. It has a
number of hoops to jump through, each of which has a well defined set
of success criteria. By contrast, performance testing to be useful
needs to be more flexible in several ways - including the code tested
(ideally, we'd be able to try out several patch sets) and the tests
performed. What is more, measurement is on a continuous scale rather
than a set of boolean flags. And the trigger to run tests needs to be
something other than changes in code on a well known branch.

Incidentally, use of a different SCM system might well make
constructing test sets more simple. Imagine, say, in SVN, you would
create a branch called "test-set-yyyy-mm-dd" or some such, make your
changes there, add a test script under some well known name, and commit
the branch. Then you might put an entry on the server that contains the
name of the branch and a date/time after which we would consider the
tests stale. The performance-farm member would periodically check with
the server for new entries, switch to the named branch, build, run the
tests, and upload the results.

If someone wants to have a go at implementing such a system, I can give
them some help and advice. Otherwise I will put it on my list of things
to do, but the earliest I could devote any time to it would be very late
this year.

cheers

andrew