Re: What constitutes "reproducible" numbers from pgbench?

On Thu, 23 Apr 2015 11:07:05 +0200
<Holger(dot)Friedrich-Fa-Trivadis(at)it(dot)nrw(dot)de> wrote:

> On Tuesday, April 21, 2015 7:43 PM, Andy Colson wrote:
> > On 4/21/2015 9:21 AM, Holger(dot)Friedrich-Fa-Trivadis(at)it(dot)nrw(dot)de wrote:
> >> Exactly what constitutes "reproducible" values from pgbench? I keep
> >> getting a range between 340 tps and 440 tps or something like that
> > I think its common to get different timings. I think its ok because things are changing (files, caches, indexes, etc).
>
> As I found out, our test server is a virtual machine, so while I should be "alone" on that virtual machine, of course I have no idea what else might be going on on the physical server the virtual machine is running on. That would explain the somewhat wide variations.
>
> Qingqing Zhou wrote that the range between 340 tps and 440 tps I keep getting is not ok and numbers should be the same within several per cent. Of course, if other things are going on on the physical server, I can't always expect a close match.
>
> Since someone asked, the point of the exercise is to see if and how various configurations in postgresql.conf are affecting performance.

You're going to have difficulty doing that sort of tuning and testing
on a VM. Even when there's nothing else going on, VMs tend to have
a wider range of behaviors than native installs (since things like
cron jobs can run both on the host and the guest OS, as well as
other reasons, I'm sure).

Whether such an endeavour is worthwhile depends on your reason for
doing it. If your production environment will also be a VM of similar
configuration to this one, then I would proceed with the tests, simply
tracking the +/- variance and keeping it in mind; since you'll likely
see the same variance on production.

If you're doing it for your own general learning, then it might still
be worth it, but it's hardly an idea setup for that kind of thing.

--
PT <wmoran(at)potentialtech(dot)com>