Re: distributed performance testing

On Sat, Aug 13, 2005 at 06:29:58PM -0400, Tom Lane wrote:
> Andrew Dunstan <andrew(at)dunslane(dot)net> writes:
> > Incidentally, use of a different SCM system might well make
> > constructing test sets more simple. Imagine, say, in SVN, you would
> > create a branch called "test-set-yyyy-mm-dd" or some such, make your
> > changes there, add a test script under some well known name, and commit
> > the branch.
>
> This seems a pretty unconvincing argument for SVN ... we could perfectly
> well do that in CVS, no?

In any case, I think it probably makes more sense to specify tests as
'pull from CVS as of this date/tag and then (optionally) apply these
patches'. It doesn't seem to make sense to clutter up CVS just to be
able to run performance tests.

In any case, I agree. I've been wondering if it makes sense to setup a
result repository for dbt* where people running dbt tests could submit
results (along with machine config, etc). ISTM that having that would be
beneficial on it's own, and we could then build an additional framework
for pushing desired tests out to a set of machines.

Of course we could use pgbench for this instead of dbt*, but ISTM that
dbt is a better choice since it's useful for a broader set of people.
The downside is it requires dbt, but that doesn't seem to be a major
issue. Also, using dbt means we can test different use cases (dbt2 ~=
TPC-C, dbt3 ~= TPC-H, etc), while pgbench is just a single benchmark.
--
Jim C. Nasby, Sr. Engineering Consultant jnasby(at)pervasive(dot)com
Pervasive Software http://pervasive.com 512-569-9461