Re: Slow queries / commits, mis-configuration or hardware issues?

On Nov 16, 2011, at 8:52 AM, Tomas Vondra wrote:

> On 16 Listopad 2011, 2:21, Cody Caughlan wrote:
>> How did you build your RAID array? Maybe I have a fundamental flaw /
>> misconfiguration. I am doing it via:
>>
>> $ yes | mdadm --create /dev/md0 --level=10 -c256 --raid-devices=4
>> /dev/xvdb /dev/xvdc /dev/xvdd /dev/xvde
>> $ pvcreate /dev/md0
>> $ vgcreate lvm-raid10 /dev/md0
>> $ lvcreate -l 215021 lvm-raid10 -n lvm0
>> $ blockdev --setra 65536 /dev/lvm-raid10/lvm0
>> $ mkfs.xfs -f /dev/lvm-raid10/lvm0
>> $ mkdir -p /data && mount -t xfs -o noatime /dev/lvm-raid10/lvm0 /data
>
> I'm not using EC2 much, and those were my first attempts with ephemeral
> storage, so this may be a stupid question, but why are you building a
> RAID-10 array on an ephemeral storage, anyway?
>
> You already have a standby, so if the primary instance fails you can
> easily failover.
>

Yes, the slave will become master if master goes down. We have no plan to try and resurrect the master in the case of failure, hence the choice of ephemeral vs EBS.

We chose RAID10 over RAID0 to get the best combination of performance and minimizing probability of a single drive failure bringing down the house.

So, yes, RAID0 would ultimately deliver the best performance, with more risk.

> What are you going to do in case of a drive failure? With a server this is
> rather easy - just put there a new drive and you're done, but can you do
> that on EC2? I guess you can't do that when the instance is running, so
> you'll have to switch to the standby anyway, right? Have you ever tried
> this (how it affects the performance etc.)?
>

As far as I know one cannot alter the ephemeral drives in a running instance, so yes, the whole instance would have to be written off.

> So what additional protection does that give you? Wouldn't a RAID-0 be a
> better utilization of the resources?
>

Too much risk.

> Tomas
>