Re: Quality and Performance

On Tue, 2007-11-27 at 13:32 -0500, Andrew Dunstan wrote:

> We also need to talk about what would be a good set of tests to run.

I think we should develop a series of performance regression tests that
can be run as an option on the buildfarm. We'd want a separate page for
that with graphs etc, as you suggest.

My vision for that is a set of tests that test very specific aspects of
code, much the same way as the regression tests attempt feature
coverage. Examples would be
- 10000 INSERTs
- 10000 INSERTs using multi-VALUEs clauses
- 100000 rows inserted by COPY
- 100000 rows inserted by CTAS

We would need a way to compare results between releases, so we can see
which aspects have regressed/improved, just as we have with the
buildfarm. That will also be food for release notes, where we can
mention all actions that are >5% faster, or anything we must regrettably
report as being slower.

Sounds like it's waiting on somebody to make the first move, so maybe I
should do that, then let everybody else chip into the framework.

Should we do this as part of core, or as a separate pgfoundry project?

--
Simon Riggs
2ndQuadrant http://www.2ndQuadrant.com