Re: Justifying a PG over MySQL approach to a project

Greg Smith <greg(at)2ndquadrant(dot)com> writes:
> They do have a regression test suite:
> http://dev.mysql.com/doc/refman/5.0/en/mysql-test-suite.html

> But it's not really clear that they run it on every platform, i.e.
> http://ourdelta.org/hidden-tests-of-the-mysql-testsuite

They definitely don't run it on every combination of allegedly-supported
options. I had to turn on --with-big-tables in the Red Hat build awhile
ago, which is probably a good thing anyway (though if so, why isn't it
default?); the reason I had to do it was the regression tests started
showing obvious failures without it, proving that they don't bother to
run any internal tests without it.

I'm not sure how thorough our buildfarm coverage is for different option
combinations, but the fact that their test suite takes circa four hours
to run is *not* an advantage for them in the comparison. They clearly
haven't got the resources to run all the cases they ought to. (BTW,
that's 4 hours for standard "make check", not any of the optional tests
referred to in the above-cited blog entry.)

> This supports the rumors I've heard that the development on the database
> regularly cheats by just disabling tests that don't work right in some
> situations, just so they can ship saying "there's no know issues!".

Oh, absolutely. They actually have a standard mechanism built into the
test harness for disabling tests that are currently failing, and the set
that are so disabled changes with every update. Compare the contents
of mysql-test/t/disabled.def in various releases sometime.

regards, tom lane