Re: PSA: we lack TAP test coverage on NetBSD and OpenBSD

>> I'd rather keep it by simply adding the "|unknown" alternative. 30 years
>> of programming have taught me that testing limit & error cases is useful,
>> although you never know when it will be proven so.
>
> Sorry, I don't buy this line of argument.

> Reasonable test design requires making cost/benefit tradeoffs: the cost
> to run the test over and over, and the cost to maintain the test itself
> (e.g. fix portability issues in it) have to be balanced against the
> probability of it finding something useful. I judge that the chance of
> this particular test finding something is small, and I've had quite
> enough of the maintenance costs.
>
> Just to point up that we're still not clearly done with the maintenance
> costs of supposing that we know how every version of getopt_long will
> spell this error message, I note that my Linux box seems to have two
> variants of it:
>
> $ pgbench -z
> pgbench: invalid option -- 'z'
> Try "pgbench --help" for more information.
> $ pgbench --z
> pgbench: unrecognized option '--z'
> Try "pgbench --help" for more information.
>
> of which the "invalid" alternative is also not in our list right now.
> Who's to say how many more versions of getopt_long are out there,
> or what the maintainers thereof might do in the future?

ISTM that the getopt implementers imagination should run out in the end:-)
invalid, unknown, unrecognized, unexpected, incorrect... Ok English has
many words:-)

>> I agree that some tests can be useless, but I do not think that it applies
>> to this one. This test also checks that under a bad option pgbench stops
>> with an appropriate 1 exit status.
>
> It's possible that it's worth the trouble to check for exit status 1,
> but I entirely fail to see the point of checking exactly what is the
> spelling of a message that is issued by code not under our control.
>
> Looking closer at the test case:
>
> [
> 'bad option',
> '-h home -p 5432 -U calvin -d --bad-option',
> [ qr{(unrecognized|illegal) option}, qr{--help.*more information} ]
> ],
>
> ISTM that just removing the first qr{} pattern, and checking only that
> we get the text that *is* under our control, is a reasonable compromise
> here.

Possibly. I'd be a little happier if it checks for a non-empty error
message, eg qr{...} or qr{option} (the message should say something about
the option).

--
Fabien.