Re: performance-test farm

Dne 11.5.2011 23:41, Kevin Grittner napsal(a):
> Tomas Vondra <tv(at)fuzzy(dot)cz> wrote:
>
>> 1) Is there something that might serve as a model?
>
> I've been assuming that we would use the PostgreSQL Buildfarm as a
> model.
>
> http://buildfarm.postgresql.org/

Yes, I was thinking about that too, but

1) A buildfarm used for regular building / unit testing IMHO may not
be the right place to do performance testing (not sure how isolated
the benchmarks can be etc.).

2) Not sure how open this might be for the developers (if they could
issue their own builds etc.).

3) If this should be part of the current buildfarm, then I'm afraid I
can't do much about it.

>> 2) How would you use it? What procedure would you expect?
>
> People who had suitable test environments could sign up to
> periodically build and performance test using the predetermined test
> suite, and report results back for a consolidated status display.
> That would spot regressions.

So it would be a 'distributed farm'? Not sure it that's a good idea, as
to get reliable benchmark results you need a proper environment (not
influenced by other jobs, changes of hw etc.).

> It would be nice to have a feature where a proposed patch could be
> included for a one-time build-and-benchmark run, so that ideas could
> be tried before commit. It can be hard to anticipate all the
> differenced between Intel and AMD, Linux and Windows, 32 bit and 64
> bit, etc.

Yes, that's one of the main goals - to allow developers to benchmark
their patches under various workloads. I don't think we'll be able to
get all those configurations, though.

>> 3) Any other features expected?
>
> Pretty graphs? :-)
Sure. And it will be Web 2.0 ready ;-)

Tomas