Re: performance-test farm

Dne 12.5.2011 16:36, Tom Lane napsal(a):
>> * Josh Berkus (josh(at)agliodbs(dot)com) wrote:
>>> The first problem is plaform performance, which would be a matter of
>>> expanding the buildfarm to include a small set of performance tests ...
>>> probably ones based on previously known problems, plus some other simple
>>> common operations. The goal here would be to test on as many different
>>> machines as possible, rather than getting full coverage of peformance.
>
> I think it's a seriously *bad* idea to expect existing buildfarm members
> to produce useful performance data. Very few of them are running on
> dedicated machines, and some are deliberately configured with
> performance-trashing options. (I think just about all of 'em use
> --enable-cassert, but there are some with worse things...)
>
> We can probably share a great deal of the existing buildfarm code and
> infrastructure, but the actual members of the p-farm will need to be a
> separate collection of machines running different builds.

Yes, I agree that to get reliable and useful performance data, we need
well defined environment (dedicated machines, proper settings) and this
probably is not possible with most of the current buildfarm members.
That's not an issue of the buildfarm - it simply serves other purposes,
not performance testing.

But I believe using the buildfarm framework is a good idea, and maybe we
could even use some of the nodes (those that are properly set).

Not sure if there should be two separate farms (one to test builds, one
to test performance), or if we could run one farm and enable
'performance-testing modules' only for some of the members.

regards
Tomas