Need some help analyzing some benchmarks

Before I deploy some new servers, I figured I would do some
benchmarking.

Server is a Dual E5620, 96GB RAM, 16 x 450GB SAS(15K) drives.

Controller is an Areca 1680 with 2GB RAM and battery backup.

So far I have only run bonie++ since each cycle is quite long (writing
192GB).

My data partition is 12 drives in RAID 1+0 (2.7TB) running UFS2.
Vfs.read_max has been set to 32, and no other tuning has been done.

Files system is not mounted with noatime at this point.

Below are the results:

db1# bonnie++ -d /usr/local/pgsql -c 4 -n 2:10000000:1000000:64 -u pgsql

Version 1.96 ------Sequential Output------ --Sequential Input-
--Random-

Concurrency 4 -Per Chr- --Block-- -Rewrite- -Per Chr- --Block--
--Seeks--

Machine Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP
/sec %CP

db1.stackdump. 192G 860 99 213731 52 28518 45 1079 70 155479 34
49.9 12

Latency 10008us 2385ms 1190ms 457ms 2152ms
231ms

Version 1.96 ------Sequential Create------ --------Random
Create--------

db1.stackdump.local -Create-- --Read--- -Delete-- -Create-- --Read---
-Delete--

files:max:min /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP
/sec %CP

2:10000000:1000000/64 49 33 128 96 277 97 57 39 130 90
275 97

Latency 660ms 13954us 13003us 904ms 334ms
13365us

Not having anything to compare it to, I do not know if these are decent
numbers or not - they are definitely slower than a similar setup which
was posted recently using XFS on Linux, but I have not found anything
in FreeBSD using UFS2 to compare it to. What strikes me in particular
is that the write performance is higher than the read performance - I
would have intuitively expected it to be the other way around.

My log partition is a RAID1, same drives. Performance follows:

Version 1.96 ------Sequential Output------ --Sequential Input-
--Random-

Concurrency 4 -Per Chr- --Block-- -Rewrite- -Per Chr- --Block--
--Seeks--

Machine Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP
/sec %CP

db1.stackdump. 192G 861 99 117023 28 20142 43 359 23 109719 24
419.5 12

Latency 9890us 13227ms 8944ms 3623ms 2236ms
252ms

Version 1.96 ------Sequential Create------ --------Random
Create--------

db1.stackdump.local -Create-- --Read--- -Delete-- -Create-- --Read---
-Delete--

files:max:min /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP
/sec %CP

2:10000000:1000000/64 24 16 121 93 276 97 22 15 134 93
275 97

Latency 4070ms 14029us 13079us 15016ms 573ms
13369us

After seeing these results, I decided to download the areca cli and
check the actual setup

Info from the RAID controller follows:

CLI> sys info

The System Information

===========================================

Main Processor : 1200MHz

CPU ICache Size : 32KB

CPU DCache Size : 32KB

CPU SCache Size : 512KB

System Memory : 2048MB/533MHz/ECC

Firmware Version : V1.48 2010-10-21

BOOT ROM Version : V1.48 2010-01-04

Serial Number : Y051CABVAR600825

Controller Name : ARC-1680

Current IP Address : 192.168.1.100

CLI> rsf info raid=2

Raid Set Information

===========================================

Raid Set Name : Raid Set # 001

Member Disks : 12

Total Raw Capacity : 5400.0GB

Free Raw Capacity : 0.0GB

Min Member Disk Size : 450.0GB

Raid Set State : Normal

CLI> vsf info vol=2

Volume Set Information

===========================================

Volume Set Name : ARC-1680-VOL#001

Raid Set Name : Raid Set # 001

Volume Capacity : 2700.0GB

SCSI Ch/Id/Lun : 00/00/01

Raid Level : Raid1+0

Stripe Size : 8K

Member Disks : 12

Cache Mode : Write Back

Tagged Queuing : Enabled

Volume State : Normal

===========================================

Having done this, I noticed that the stripe size is configured to 8K.

I am thinking the problem may be due to the stripe size. I had asked
the vendor to set up the file system for these two arrays with 8K
blocks, and I believe they may have misunderstood my request and set the
stripe size to 8K. I assume increasing the stripe size will improve the
performance.

What stripe sizes are you typically using? I was planning on setting it
up with a 64K stripe size.

TIA,

Benjamin