Re: Having I/O problems in simple virtualized environment

Op 30-01-12 02:52, Jose Ildefonso Camargo Tolosa schreef:
> On Sun, Jan 29, 2012 at 6:18 PM, Ron Arts <ron(dot)arts(at)gmail(dot)com> wrote:
>> Hi list,
>>
>> I am running PostgreSQL 8.1 (CentOS 5.7) on a VM on a single XCP (Xenserver) host.
>> This is a HP server with 8GB, Dual Quad Core, and 2 SATA in RAID-1.
>>
>> The problem is: it's running very slow compared to running it on bare metal, and
>> the VM is starving for I/O bandwidht, so other processes (slow to a crawl.
>> This does not happen on bare metal.
>>
>> I had to replace the server with a bare-metal one, I could not troubleshoot in production.
>> Also it was hard to emulte the workload for that VM in a test environment, so I
>> concentrated on PostgreSQLand why it apparently generated so much I/O.
>>
>> Before I start I should confess having only spotty experience with Xen and PostgreSQL
>> performance testing.
>>
>> I setup a test Xen server created a CentOS5.7 VM with out-of-the-box PostgreSQL and ran:
>> pgbench -i pgbench ; time pgbench -t 100000 pgbench
>> This ran for 3:28. Then I replaced the SATA HD with an SSD disk, and reran the test.
>> It ran for 2:46. This seemed strange as I expected the run to finish much faster.
>>
>> I reran the first test on the SATA, and looked at CPU and I/O use. The CPU was not used
>> too much in both the VM (30%) and in dom0 (10%). The I/O use was not much as well,
>> around 8MB/sec in the VM. (Couldn't use iotop in dom0, because of missing kernel support
>> in XCP 1.1).
>>
>> It reran the second test on SSD, and experienced almost the same CPU, and I/O load.
>>
>> (I now probably need to run the same test on bare metal, but didn't get to that yet,
>> all this already ruined my weekend.)
>>
>> Now I came this far, can anybody give me some pointers? Why doesn't pgbench saturate
>> either the CPU or the I/O? Why does using SSD only change the performance this much?
>
> Ok, one point: Which IO scheduler are you using? (on dom0 and on the VM).

Ok, first dom0:

For the SSD (hda):
# cat /sys/block/sda/queue/scheduler
[noop] anticipatory deadline cfq

For the SATA:
# cat /sys/block/sdb/queue/scheduler
noop anticipatory deadline [cfq]

Then in the VM:

# cat /sys/block/xvda/queue/scheduler
[noop] anticipatory deadline cfq

Ron