Re: PSA: we lack TAP test coverage on NetBSD and OpenBSD

Hello Tom,

>> Given these results, I do not think that it is useful to change
>> random_zipfian TAP test parameter from 2.5 to something else.
>
> I'm not following this argument. The test case is basically useless
> for its intended purpose with that parameter, because it's highly
> likely that the failure mode it's supposedly checking for will be
> masked by the "random" function's tendency to spit out the same
> value all the time.

The first value is taken about 75% of the time for N=1000 and s=2.5, which
means that a non deterministic implementation would succeed about 0.75² ~
56% of the time on that one. Then there is other lower probability random
successes. ISTM that if a test fails every three run it would be detected,
so the purpose of testing random_zipfian determinism is somehow served.

Also, the drawing procedure is less efficient when the parameter is close
to 1 because it is more likely to loop, and there are other values tested,
0.5 and 1.3 (note that the code has two methods, depending on whether the
parameter is below or above 1), so I think that having something different
is better.

If you want something more drastic, using 1.5 instead of 2.5 would reduce
the probability of accidentaly passing the test by chance to about 20%, so
it would fail 80% of the time.

> We might as well drop zipfian from the test altogether and save
> ourselves some buildfarm cycles.

All 4 random functions are tested together on the same run, removing a
particular one does not seem desirable to me.

--
Fabien.