Re: [HACKERS] Re: [COMMITTERS] pgsql: Remove pgbench "progress" test pending solution of its timing is (fwd)

On Sun, Sep 24, 2017 at 11:30 PM, Fabien COELHO <coelho(at)cri(dot)ensmp(dot)fr> wrote:
> Attached is an attempt at improving the situation:
>
> (1) there are added comments to explain the whys, which is to provide
> coverage for pgbench time-related features... while still not being
> time-sensitive, which is a challenge.
>
> (2) the test now only expects "progress: \d" from the output, so it is
> enough that one progress is shown, whenever it is shown.
>
> (3) if the test is detected to have gone AWOL, detailed log checks are
> coldly skipped.
>
> This would have passed on "skink" under the special conditions it
> encountered.
>
> I cannot guaranty that it would pass under any circumstance, though.
>
> If it still encounters a failure, ISTM that it should only be a missing
> "progress:" in the output, which has not been encountered so far.
>
> If it occurs, a few options would remain, none of them very convincing:
>
> - give the test some more time, eg 3 seconds (hmmm... could still fail
> after any time...)
>
> - drop the "progress:" expectation (hmmm... but then it does not check
> anything).
>
> - make the "progress:" output check conditional to the running time
> (hmmm... it would require changing the command_checks_all function,
> and there is still a chance that the bench was stuck for 2 seconds
> then given time to realize that it has to stop right now...)
>
> - use an even simpler transaction, eg "SELECT;" which is less likely to
> get stuck (but still could get stuck...).
>
> For the random-ness related test (--sample-rate), we could add a feature to
> pgbench which allows to control the random seed, so that the number of
> samples could be known in advance for testing purposes.

This didn't get any reviews, so bumped to CF 2018-01.
--
Michael