Re: [OT] RAID controllers blocking one another?

Sean Davis wrote:
> We are PURE amateurs, so that is possible. We are running 12-disk RAID6
> (750GB drives--this might be a difference from what you are thinking)
> partitions under LVM. We have write-cache enabled and are using ext3
> filesystems. Any suggestions?

We use the 9650 controllers in RAID 6 a lot. With 12 discs you should be
able to get about 550MB/sec sequential write and 680MB/sec read easily.
But it's already pretty clear to me why you can't get anywhere near that:

The first thing is that ext3 is a file system designed for small block
devices that have a single physical head to seek -- not for large arrays
that consist of multiple physical devices that can seek independently.
Especially this is true for the anticipatory I/O scheduler that is used
in conjunction with ext3. Additionally ext3 does not know any form of
stripe alignment which is an absolute requirement for arrays that use a
parity based RAID level (RAID 5/6). If you don't stripe align your file
system to the stripe width configured in your controller, you will
suffer massive I/O trashing. Your controller and discs will be busy with
Load-XOR-Store cycles. That means that your discs will do a ton of micro
seeks on every write instead of just writing the chunk of data out. Both
performance and lifespan of your discs will suffer for no reason.

My recommendation is either Linux XFS or Solaris ZFS. Both are designed
form the ground up for such setups. If this box is not yet in production
and you like to test something I can give you a fitting mkfs.xfs line
for your setup -- the defaults won't work well.

The last thing to note is that LVM will mess with any kind of stripe
alignment, so it's quite useless for parity based arrays. If you need
serious volume management look into ZFS.

Finally you might need to do tweak some scheduler params, depending if
you chose to go with deadline or CFQ. Deadline is recommend by 3ware and
usually the best choice for PostgreSQL as well. Speaking of PostgreSQL,
put $PGDATA on a RAID 10 array, it will suffer on RAID 5/6.

--
Best regards,
Hannes Dorbath