Re: (download ANSI SQL benchmark?) Re: Postgres article

At 10:19 AM 11/21/00 +0000, Pete Forman wrote:
>Don Baccus writes:
> > I also hope that the PG crew, and Great Bridge, never stoop so low
> > as to ship benchmarks wired to "prove" PG's superiority.
>
>I thought that Great Bridge's August benchmarks were rather skewed.
>They only used one particular test from the AS3AP suite. That was the
>basis for their headline figure of 4-5 times the performance of the
>competition.
>
>I was however impressed by the TPC-C results. MySQL and Interbase
>were unable to complete them. PostgreSQL showed almost identical
>performance over a range of loads to Proprietary 1 (version 8.1.5, on
>Linux) and Proprietary 2 (version 7.0, on NT).

Great Bridge didn't do the benchmarking, they hired a third party to
do so. And that third party didn't, AFAIK, cherry-pick tests in order
to "prove" PG's superiority.

The report itself mentioned the testing group's surprise over MySQL's
poor showing in the simple, non-TPC-C test. I'm sure it was tossed
in so they could answer the question "how much does it cost you to
use a transaction-based system rather than MySQL", since avoiding that
overhead is the big argument that the MySQL makes in favor of their
product. I'm sure the hope was there that the answer would be "not all
that much", instead the answer was "gee, you're not that fast after
all".

Clearly the real target of the benchmark effort was Oracle. However
inadequate the benchmarking effort might've been (they're all inadequate,
after all) the fact is that Great Bridge at least did run a set of
standard benchmarks.

The MySQL folk have always cherry-picked their benchmarks, long lied
about features in PG, do their benchmarking using default values
for PG's shared buffer etc WITHOUT TELLING PEOPLE while at the same
time installing MySQL with larger-than-default memory usage limits (the
group hired by GB used MySQL's default installation, but EXPLICITLY SAID
SO in the report), etc.

The GB-financed benchmarks weren't perfect, but they weren't dishonest.
The MySQL folks have done things over the years that have been out-and-out
dishonest, IMO.

- Don Baccus, Portland OR <dhogaza(at)pacifier(dot)com>
Nature photos, on-line guides, Pacific Northwest
Rare Bird Alert Service and other goodies at
http://donb.photo.net.