Re: pgsql: Remove pgbench "progress" test pending solution of its timing is

Fabien COELHO <coelho(at)cri(dot)ensmp(dot)fr> writes:
> So this means somehow giving up on coverage, because of one host which can
> fail under very unusual testing conditions once in a while.

You are attacking a straw man. There is no reason whatever to believe
that the problem is confined to one host. We have quite a number of slow
buildfarm hosts: some are using CLOBBER_CACHE_ALWAYS, some are overloaded,
some are just plain slow. Also, regression test failures outside the
context of our buildfarm are just as bad if not worse. I still remember
the pain I had with mysql's regression tests, which routinely fell over
in Red Hat's package buildfarm even though they always succeeded on a
personal workstation with nothing else going on. I do not want our own
packagers to have that kind of experience with Postgres.

> I would prefer to keep the test and have a warning instead, something
> like "ok, although a test which is allowed to rarely fail failed", but at
> least the feature are tested most of the time.

If the failures are ignored, how much of a test have you really got?
Nobody inquires into nominally-passed buildfarm runs to see what
actually happened.

In the TAP-test context, you do have quite a bit of freedom to decide
what is a pass and what is a fail. So maybe the path forward is to be
more flexible about the pass conditions for this test. But I can't
help thinking that these results have exposed some outright bugs too.

regards, tom lane