Re: performance-test farm

Dne 12.5.2011 00:21, Kevin Grittner napsal(a):
> Tomas Vondra <tv(at)fuzzy(dot)cz> wrote:
>> Dne 11.5.2011 23:41, Kevin Grittner napsal(a):
>>> Tomas Vondra <tv(at)fuzzy(dot)cz> wrote:
>>>
>>>> 1) Is there something that might serve as a model?
>>>
>>> I've been assuming that we would use the PostgreSQL Buildfarm as
>>> a model.
>>>
>>> http://buildfarm.postgresql.org/
>>
>> Yes, I was thinking about that too, but
>>
>> 1) A buildfarm used for regular building / unit testing IMHO may
>> not be the right place to do performance testing (not sure how
>> isolated the benchmarks can be etc.).
>
> I'm not saying that we should use the existing buildfarm, or expect
> current buildfarm machines to support this; just that the pattern of
> people volunteering hardware in a similar way would be good.

Good point. Actually I was not aware of how the buildfarm works, all I
knew was there's something like that because some of the hackers mention
a failed build on the mailing list occasionally.

So I guess this is a good opportunity to investigate it a bit ;-)

Anyway I'm not sure this would give us the kind of environment we need
to do benchmarks ... but it's worth to think of.

>
>> 2) Not sure how open this might be for the developers (if they
>> could issue their own builds etc.).
>
> I haven't done it, but I understand that you can create a "local"
> buildfarm instance which isn't reporting its results. Again,
> something similar might be good.

Well, yeah. So the developers would get a local 'copy' of all the
benchmarks / workloads and could run them?

>> 3) If this should be part of the current buildfarm, then I'm
>> afraid I can't do much about it.
>
> Not part of the current buildfarm; just using a similar overall
> pattern. Others may have different ideas; I'm just speaking for
> myself here about what seems like a good idea to me.

OK, got it.

>>>> 2) How would you use it? What procedure would you expect?
>>>
>>> People who had suitable test environments could sign up to
>>> periodically build and performance test using the predetermined
>>> test suite, and report results back for a consolidated status
>>> display. That would spot regressions.
>>
>> So it would be a 'distributed farm'? Not sure it that's a good
>> idea, as to get reliable benchmark results you need a proper
>> environment (not influenced by other jobs, changes of hw etc.).
>
> Yeah, accurate benchmarking is not easy. We would have to make sure
> people understood that the machine should be dedicated to the
> benchmark while it is running, which is not a requirement for the
> buildfarm. Maybe provide some way to annotate HW or OS changes?
> So if one machine goes to a new kernel and performance changes
> radically, but other machines which didn't change their kernel
> continue on a level graph, we'd know to suspect the kernel rather
> than some change in PostgreSQL code.

I guess we could run a script that collects all those important
parameters and then detect changes. Anyway we still need some 'really
stable' machines that are not changed at all, to get a long-term baseline.

But I guess that could be done by running some dedicated machines ourselves.

regards
Tomas