Re: Quality and Performance

Tom Lane wrote:
> Andrew Dunstan <andrew(at)dunslane(dot)net> writes:
>> Josh Berkus wrote:
>>> ... DW operations aren't
>>> really testable without 18 hours to generate data ... but we could test a
>>> lot of things.
>
>> Performance isn't just about humungous DW apps.
>
> Indeed. I think the real take-home lesson from these past few days'
> discussion is that *any* particular view of performance is going to
> miss things that don't affect that case, but do affect somebody else.

yep - but the "do not affect somebody else" might be one that could be
actually catched by something like a benchfarm.

>
> What I find most worrisome about the notion of setting up a
> performance-farm is that it will encourage us to optimize with blinkers
> on --- that is, that we will consider only the specific cases measured
> by whatever tests are included in the farm, and will happily pessimize
> other cases. We can ameliorate that a bit if we can get a sufficiently
> wide variety of test cases, but it will always be a concern. And
> dogmatic positions like "only cases involving terabytes of data are
> worth testing" are definitely not going to help.

agreed - I don't think having the tests itself in core (at least
initially) is such a good idea(neither am I sure tacking it on top of
the buildfarm really is).
There are a LOT of things we could do with such a farm/infrastructure
but it will take time to exactly figure out what we can reasonably do on
an automated/regular base and in a common framework.

Stefan