how to demonstrate the effect of direct I/O ?

Hi all,

I've been running a lot of benchmarks recently (I'll publish the results
once I properly analyze them). One thing I'd like to demonstrate is the
effect of direct I/O when the wal_fsync_method is set to
open_sync/open_datasync.

I.e. I'd like to see cases when this improves/hurts performance
(compared to fsync/fdatasync) and if/how this works on SSD compared to
old-fashioned HDD. But no matter what, I see no significant differences
in performance.

This is what pg_test_fsync gives on the SSD (Intel 320):

open_datasync 12492.192 ops/sec
fdatasync 11646.257 ops/sec
fsync 9839.101 ops/sec
fsync_writethrough n/a
open_sync 10420.971 ops/sec

and this is what I get on the HDD (7.2k SATA)

open_datasync 120.041 ops/sec
fdatasync 120.042 ops/sec
fsync 48.000 ops/sec
fsync_writethrough n/a
open_sync 48.116 ops/sec

I can post the rest of the pg_test_fsync output if needed.

What should I do to see the effect of direct I/O? I'm wondering if I
need something like a RAID array or a controller with write cache to see
the difference.

All this was run on a kernel 3.1.5 using an ext4 filesystem.

thanks
Tomas