Re: Does PostgreSQL have complete functional test cases?

On 11/09/2012 10:36 AM, Alvaro Herrera wrote:
> Craig Ringer wrote:
>
>> It'd be nice to split the tests up into clearer groups - "will fail if
>> planner settings are changed; WARNING", "will fail only if incorrect
>> result is returned; FATAL" etc. Right now, AFAIK that hasn't been done.
> Not sure that's enough of an improvement. Really, these tests should
> pass whatever the optimizer parameters are set to; instead of checking
> the exact order of the result set, they should check that the right
> tuples are returned. Except, of course, in those cases that specify
> ORDER BY.
Except for cases where the test is verifying that the optimizer chooses
the right plan. Those must fail if different optimizer inputs are
provided. OTOH, maybe those tests should explicitly set the required
optimizer parameters, rather than assuming that the PostgreSQL install's
defaults are what the test wants.

For tests without ORDER BY it'd certainly be nice to sort the output and
the expected result and *then* compare, so simple ordering differences
due to non-deterministic ordering are ignored. That isn't really
possible when the regression tests are just diffs between expected/ and
actual output, though.

--
Craig Ringer