Re: Does PostgreSQL have complete functional test cases?

On Fri, Nov 9, 2012 at 10:50:42AM +0800, Craig Ringer wrote:
> On 11/09/2012 10:36 AM, Alvaro Herrera wrote:
> > Craig Ringer wrote:
> >
> >> It'd be nice to split the tests up into clearer groups - "will fail if
> >> planner settings are changed; WARNING", "will fail only if incorrect
> >> result is returned; FATAL" etc. Right now, AFAIK that hasn't been done.
> > Not sure that's enough of an improvement. Really, these tests should
> > pass whatever the optimizer parameters are set to; instead of checking
> > the exact order of the result set, they should check that the right
> > tuples are returned. Except, of course, in those cases that specify
> > ORDER BY.
> Except for cases where the test is verifying that the optimizer chooses
> the right plan. Those must fail if different optimizer inputs are
> provided. OTOH, maybe those tests should explicitly set the required
> optimizer parameters, rather than assuming that the PostgreSQL install's
> defaults are what the test wants.
>
> For tests without ORDER BY it'd certainly be nice to sort the output and
> the expected result and *then* compare, so simple ordering differences
> due to non-deterministic ordering are ignored. That isn't really
> possible when the regression tests are just diffs between expected/ and
> actual output, though.

I question if the regression tests really warrant this kind of detailed
work. Not only is it work that could be applied somewhere else, but if
it makes adding new tests harder, it might be a net loss.

--
Bruce Momjian <bruce(at)momjian(dot)us> http://momjian.us
EnterpriseDB http://enterprisedb.com

+ It's impossible for everything to be true. +