Re: [HACKERS] Re: [COMMITTERS] pgsql: Remove pgbench "progress" test pending solution of its timing is (fwd)

Fabien COELHO <coelho(at)cri(dot)ensmp(dot)fr> writes:
>> It does not look like the remainder of this patch is going to be committed
>> and I don't think it makes sense to keep moving the patch indefinitely.
>> Unless something changes by the end of this CF I'll mark it Returned With
>> Feedback.

> I'd be rather unclear about what the actual feedback is, though. I'd
> interpret it as "pg does not care much about code coverage". Most clients
> are in the red on coverage.postgresql.org. I'd like pgbench at least to be
> in the green, but it does not look that it will ever be the case.

The reason why the first iteration failed was that it was insufficiently
insensitive to timing. So the problem for a replacement patch is
"how do you test fundamentally timing-sensitive behavior in a
timing-insensitive way?". It's not really clear to me that that's
possible, so I don't have a lot of faith that this patch wouldn't fail
as well.

I'm also a bit disturbed that the patch seems to be changing pgbench's
behavior for no reason other than to try to make the test produce the
same results no matter what the actual machine performance is ... which
seems, at best, rather contrary to pgbench's real purpose. So I wonder,
are those behavioral changes a net win from a user's standpoint? If not
we're letting the tail wag the dog. (If they are a win, they ought to
be submitted and defended independently, and maybe there ought to be
some documentation changes alongside.)

I'm not against having better test coverage numbers, but it's a means
to an end not an end in itself. It has to be balanced against the
amount of effort to be put into testing (as opposed to actually
improving our software).

regards, tom lane