Jonah H. Harris wrote:

On Sun, Dec 14, 2008 at 11:05 PM, justin <justin@emproshunts.com> wrote:

I would like to see them even if PostgreSql comes dead last.


Well, running an official TPC benchmark is an interesting process.
First, you have to have a spec-compliant benchmark kit, which we
don't.  Second, you have to work with a hardware vendor, and none of
the vendors are going to work with you if you can't make their
hardware look good; they don't want to come in last even if it's due
to your software.  I know, I talked to three of the major hardware
vendors.  Lastly, it costs a fair amount of money and time to perform
an audited benchmark, which I don't think the community wants to pay
for.

Thats a problem with testing. Its not cheap or easy. I do this all the time with electrical equipment. The company I work for has 20K in a environmental testing chamber, 55K in power supplies 60K in DMMs which must be certified to NIST costing another 10K every year. This is only for the lab and QC.

So yes i know testing is very expensive but it can be done and should be done independently of the Hardware/Software manufactures. If not how are we to know that this hardware/Software is really a production item. Example of independent lab is Consumer Reports, created to test products without the manufactures influence over the net result as the manufactures were hoisting bogus testing reports on the public.

The more i learn how the tests are run the more suspicious i become its totally black bagged. The test method must be easy to understand be realistic to a real model that anyone can run with X time and X money on hand. Since you have stated that is near impossible to do, i would say the testing methods need to be rethought.

Any one can test our product if they went out spend 40K on electrical lab equipment and has a basic understand of electrical principles. There is no need to have an understanding of ISO 17025 http://en.wikipedia.org/wiki/ISO_17025

Does that not prove the point Oracle is over price software compared to the
competition.


That depends on whether you need the performance or not.

That don't make sense???? Its all about cost if the other software is cheaper per transaction means you can spend the same amount of money as one would on Oracle on lets say MSSQL then blow by Oracle's performance

The only way to verify the results is duplicate the test and given these
servers cost from $100K to $10million  its unlikely these test are verified
by a 3rd party running independent hardware.  From where i come from for
result to be proven  someone else must duplicate the results independently
with only instructions  given by the original tester.


Perhaps you should download a full disclosure report of a TPC
benchmark.  First of all, they're all audited.  Second of all, every
configuration detail is provided for you to be able to run the test
yourself.  In fact, if you have a good enough reason to do so (i.e.
looking to buy a good amount of software), both Oracle and Microsoft
will give you copies of their benchmark kits for evaluation purposes

Used to be work under ISO 9000 writing standards and dealing with Auditors. I know how auditing works and ways to make yourself look good and in compliance when its completely penciled wiped into shape. Given no 3rd party does completely independent testing i view it as an all around joke.

This is the reason we chose to drop ISO certification as it had nothing to do about doing the job right