Re: [HACKERS] (download ANSI SQL benchmark?) Re: Postgres article

Don Baccus <dhogaza(at)pacifier(dot)com> writes:
> Great Bridge didn't do the benchmarking, they hired a third party to
> do so. And that third party didn't, AFAIK, cherry-pick tests in order
> to "prove" PG's superiority.

In fairness, the third party was Xperts Inc, who have long done a lot
of programming-related work for Landmark Communications; so there's a
pretty close working relationship, it's not exactly arms-length.

I think what may be more worth noting is that that benchmarking project
was started as part of Landmark's "due diligence" investigation while
deciding whether they wanted to bet a company on Postgres. They didn't
go into it with the notion of proving Postgres superior; they went into
it to find out if they were betting on a dog. They were very pleasantly
surprised (as was the core group, when we first saw the results!).
Naturally, their marketing guys said "hey, let's clean this up and
publish it". There's a certain amount of after-the-fact selection here,
since you'd certainly never have seen the results if they hadn't been
favorable to Postgres; but there was no attempt to skew the results in
Postgres' favor. If anything, the opposite.

> The MySQL folk have always cherry-picked their benchmarks, long lied
> about features in PG, do their benchmarking using default values
> for PG's shared buffer etc WITHOUT TELLING PEOPLE while at the same
> time installing MySQL with larger-than-default memory usage limits (the
> group hired by GB used MySQL's default installation, but EXPLICITLY SAID
> SO in the report), etc.

The revised results that are on GB's site now include curves for MySQL
*with* tuning per advice from the MySQL folk.

regards, tom lane