Re: [COMMITTERS] pgsql: Remove pgbench "progress" test pending solution of its timing is (fwd)

<Bounced from commits to have it on hackers>

Hello Tom,

> Well, I think it's mostly about valgrind making everything really slow. Since
> we have seen some passes from skink recently, perhaps there was also a
> component of more-load-on-the-machine-than-usual. But in the end this is
> just evidence for my point that regression tests have to be very much not
> timing-sensitive. We run them under all kinds of out-of-the-ordinary stress.

Attached is an attempt at improving the situation:

(1) there are added comments to explain the whys, which is to provide
coverage for pgbench time-related features... while still not being
time-sensitive, which is a challenge.

(2) the test now only expects "progress: \d" from the output, so it is enough
that one progress is shown, whenever it is shown.

(3) if the test is detected to have gone AWOL, detailed log checks are
coldly skipped.

This would have passed on "skink" under the special conditions it encountered.

I cannot guaranty that it would pass under any circumstance, though.

If it still encounters a failure, ISTM that it should only be a missing
"progress:" in the output, which has not been encountered so far.

If it occurs, a few options would remain, none of them very convincing:

- give the test some more time, eg 3 seconds (hmmm... could still fail
after any time...)

- drop the "progress:" expectation (hmmm... but then it does not check
anything).

- make the "progress:" output check conditional to the running time
(hmmm... it would require changing the command_checks_all function,
and there is still a chance that the bench was stuck for 2 seconds
then given time to realize that it has to stop right now...)

- use an even simpler transaction, eg "SELECT;" which is less likely to
get stuck (but still could get stuck...).

For the random-ness related test (--sample-rate), we could add a feature to
pgbench which allows to control the random seed, so that the number of samples
could be known in advance for testing purposes.

--
Fabien.