Re: Regression test fails when BLCKSZ is 1kB

Martijn van Oosterhout napsal(a):
> On Tue, Apr 22, 2008 at 10:31:53AM +0200, Zdenek Kotala wrote:
>> When you are able detect ordering difference you are able also check if it
>> is important for the test or not without any extra effort. Only what we
>> need is put some flag to test that order is not important.
>
> Not true. Sorting the file is going jumble all the results together.
> Since we perform many tests in one file, you're not going to be able to
> seperate them.

Each statement result must be sort separately, otherwise it should hide
problems. For example one statement return A instead of B and second returns B
instead of A. When sort will be used on whole file then it will be reported as a
ordering problem.

>
>> Regression test MUST BE bulletproof. If you get a error you must know that
>> it is really error (in postgresql or regtest) and must be fixed. When you
>> start to ignore some errors because it can happen sometimes you fall in the
>> trap soon.
>
> I think people are misunderstanding. You posted a bunch of diffs with
> that comment that they *appeared* to only be ordering differences. How
> good did you check? If an 8 become a 9 chances are you'd miss it.
> Having a second test checking the sorted results would at least
> preclude the chance that there really is something wrong.
>
> It was a guide, not a way of getting out of tests.
>
> Have a nice day,