Re: [HACKERS] [9.4 CF] Free VMs for Reviewers & Testers

On 07/09/2013 08:35 AM, Josh Berkus wrote:
> Since these are cloud servers, they won't work well for performance
> testing.

I did some work on that a while ago, and found that I was able to get
_astonishingly_ stable performance results out of EC2 EBS instances
using provisioned IOPS volumes. Running them as "EBS Optimized" didn't
seem to be required for the workloads I was testing on.

What isn't made very clear in the EBS provisioned I/O docs is that for
PIOPS volumes the provisioned I/O count is both a minimum and a limit;
it's a *target*, not just a guaranteed minimum. This is really handy for
testing.

For a 500 iops volume I consistently got 500 +- 5 in pg_test_fsync.

Real world performance was a bit more variable - my pgbench runs were
stable within 5-10% over 5 minute runs with a forced CHECKPOINT first.
They settled considerably over longer runs and bigger
checkpoint_segments, though, and I see the same kind of short-term
jitter in pgbench on real hardware.

So - don't write cloud hosts off for benchmarking. With proper QoS
they're actually really very good. While they're somewhat costly, being
able to spawn them for a day's runs and destroy them at the end is quite
handy, as is being able to easily automate their setup.

For testing of logical changeset streaming replication / BDR performance
I was using scripted sets of EC2 c1.medium instances with 500 piops
'io1' EBS volumes and found it to be an exceedingly useful way of
measuring relative performance.

Results were also pretty consistent across multiple launches of the same
VM, so multiple people should be able to compare results obtained with
different launches of the same VM type and configuration.

These VMs aren't well suited to vertical scaling performance tests and
pushing extremes, but they're really, really good for "what impact does
this patch have on regular real-world workloads".

Just remember to run pgbench from a different vm within the same
availability zone (with appropriate security group setup) if you try to
use EC2 for this sort of thing.

Use of instance store or regular EBS volumes will get you performance
that's absolutely all over the place, of course; only the provisioned
I/O feature seems to be any good for perf testing.

--
Craig Ringer http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Training & Services