Re: Other benchmark than OSDB

On Thu, Sep 19, 2002 at 12:35:05PM -0700, Mark Wong wrote:
> On Tue, 17 Sep 2002 12:30:25 -0700, Andrew Sullivan wrote:

> > I'm afraid that the OSDL benchmark has a (IMNSHO stupid) handicap in
> > getting it working and making it a well-respected benchmark:
>
> There is no handicap, it just needs to be done.

My point was that the handicap is one having to do with restrictions:
certain kinds of people (like my bosses) are unlikely to fund the
work if they can't use it commercially afterwards.

> "Commercial use" can be generalized to mean that you cannot use our test
> kits to promote sales for your product.

That's still hopelessly vague. I have a piece of software. I use
Postgres (or SAPdb, for that matter). I want to prove that my
free-software-backed product is just as good as anything else. So I
want to publish some benchmarks. According to the above, I can't.

So, the benchmarks may be used, I guess, for internal evaluation
only. I guess a shop that does a lot of that will find a reason to
port these benchmarks to other database engines. But by restricting
the use of the benchmarks for advertising use, the restriction
reduces the pool of potential contributors.

It's not my project, of course, and I don't want to tell anyone else
what to do. I apologise, because re-reading my original message, I
think it reads as though I was saying, "The project should have
such-and-this license." If I want such a project, I can start it
myself.

Mostly, I'm frustrated because while the commercial guys have plenty
of money to go and run 10 gazillion TPC tests, the free guys don't
have that money. I'd like to find a way to overcome that. The OSDB
is nice, but lost of people have reservations about AS3AP. So a
TPC-inspired test would be great.

> I hope it is understood that there are legal ramifications
> regarding comparing results from our test kits to TPC results.

Naturally; but it's an uphill battle trying to supplant the TPC
benchmarks, which really are an industry standard.

> I do not follow the jump from restrictions for commercial use to
> the writing of case studies. For example, there are no issues for
> using our test kits to write a case study that surveys the
> performance of various open-source databases.

Only, it would seem, if you don't then go on to argue, "Our product
is based on [some fast free database], as proven by the test kit. So
buy our product." That seems to be straightforwardly prohibited,
since it is certainly commercial use.

> > I'm sure my boss would _love_ for me to work on something we can't ever
> > use. Sheesh.
>
> What exactly do you want to do it for? I can certainly verify if it's
> possible or not.

It would be very nice to have some solid, well-regarded benchmarks to
put in our materials about why our registry software is fast. I
already know PostgreSQL competes in the field. What I need to do is
convince manager types that PostgreSQL is a good product. OSDB is
one such benchmark. Having more than one would be even better.

A
--
----
Andrew Sullivan 204-4141 Yonge Street
Liberty RMS Toronto, Ontario Canada
<andrew(at)libertyrms(dot)info> M2P 2A8
+1 416 646 3304 x110