Re: pgbench - implement strict TPC-B benchmark

"Jonah H. Harris" <jonah(dot)harris(at)gmail(dot)com> writes:
> On Wed, Jul 31, 2019 at 10:10 AM Fabien COELHO <coelho(at)cri(dot)ensmp(dot)fr> wrote:
>> I agree that nobody really cares about TPC-B per se. The point of this
>> patch is to provide a built-in example of recent and useful pgbench
>> features that match a real specification.

> I agree with this. When I was at EnterpriseDB, while it wasn't audited, we
> had to develop an actual TPC-B implementation because pgbench was too
> different. pgbench itself isn't that useful as a benchmark tool, imo, but
> if we have the ability to make it better (i.e. closer to an actual
> benchmark kit), I think we should.

[ shrug... ] TBH, the proposed patch does not look to me like actual
benchmark kit; it looks like a toy. Nobody who was intent on making their
benchmark numbers look good would do a significant amount of work in a
slow, ad-hoc interpreted language. I also wonder to what extent the
numbers would reflect pgbench itself being the bottleneck. Which is
really the fundamental problem I've got with all the stuff that's been
crammed into pgbench of late --- the more computation you're doing there,
the less your results measure the server's capabilities rather than
pgbench's implementation details.

In any case, even if I were in love with the script itself, we cannot
commit something that claims to be "standard TPC-B". It needs weasel
wording that makes it clear that it isn't TPC-B, and then you have a
problem of user confusion about why we have both not-quite-TPC-B-1
and not-quite-TPC-B-2, and which one to use, or which one was used in
somebody else's tests.

I think if you want to show off what these pgbench features are good
for, it'd be better to find some other example that's not in the
midst of a legal minefield.

regards, tom lane