Re: Does PostgreSQL have complete functional test cases?

Bruce Momjian wrote:
> On Fri, Nov 9, 2012 at 10:50:42AM +0800, Craig Ringer wrote:

> > > Not sure that's enough of an improvement. Really, these tests should
> > > pass whatever the optimizer parameters are set to; instead of checking
> > > the exact order of the result set, they should check that the right
> > > tuples are returned. Except, of course, in those cases that specify
> > > ORDER BY.
> > Except for cases where the test is verifying that the optimizer chooses
> > the right plan. Those must fail if different optimizer inputs are
> > provided.

Of course.

> > OTOH, maybe those tests should explicitly set the required
> > optimizer parameters, rather than assuming that the PostgreSQL install's
> > defaults are what the test wants.

That doesn't sound to me as good. With what I propose (not that I'm
volunteering to implement it) you could test the optimizer just by
running the tests with varying parameters, and ensuring the resultsets
are always the same.

> > For tests without ORDER BY it'd certainly be nice to sort the output and
> > the expected result and *then* compare, so simple ordering differences
> > due to non-deterministic ordering are ignored. That isn't really
> > possible when the regression tests are just diffs between expected/ and
> > actual output, though.
>
> I question if the regression tests really warrant this kind of detailed
> work. Not only is it work that could be applied somewhere else, but if
> it makes adding new tests harder, it might be a net loss.

I see this issue as similar to the isolation test framework. There is
no way that the regression test framework could have been used to
implement those tests; yet the new tests were necessary. Are isolation
tests specs harder to write than plain regression tests? Sure, they are
somewhat harder. Could that work have been applied elsewhere? Of
course (but that choice was Kevin and Heikki's prerogative). Has it
been worth their effort? You bet.

If somebody shows up that wants to implement such a thing, I wouldn't
stop them.

--
Álvaro Herrera http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Training & Services