Re: What constitutes "reproducible" numbers from pgbench?

On 4/23/2015 4:07 AM, Holger(dot)Friedrich-Fa-Trivadis(at)it(dot)nrw(dot)de wrote:
> On Tuesday, April 21, 2015 7:43 PM, Andy Colson wrote:
>> On 4/21/2015 9:21 AM, Holger(dot)Friedrich-Fa-Trivadis(at)it(dot)nrw(dot)de wrote:
>>> Exactly what constitutes "reproducible" values from pgbench? I keep
>>> getting a range between 340 tps and 440 tps or something like that
>> I think its common to get different timings. I think its ok because things are changing (files, caches, indexes, etc).
>
> Qingqing Zhou wrote that the range between 340 tps and 440 tps I keep getting is not ok and numbers should be the same within several per cent. Of course, if other things are going on on the physical server, I can't always expect a close match.
>

I disagree. Having a reproducible test withing a few percent is a great
result. But any result is informative. You're tests tell you an upper
and lower bound on performance. It tells you to expect a little
variance in your work load. It probably tells you a little about how
your vm host is caching writes to disk. You are feeling the pulse of
your hardware. Each hardware setup has its own pulse, and understanding
it will help you understand how it'll handle a load.

-Andy