Why is checkpoint so costly?

Folks,

Going over some performance test results at OSDL, our single greatest
performance issue seems to be checkpointing. Not matter how I fiddle
with it, checkpoints seem to cost us 1/2 of our throughput while they're
taking place. Overally, checkpointing costs us about 25% of our
performance on OLTP workloads.

Example: http://khack.osdl.org/stp/302671/results/0/

Can we break down everything that happens during a checkpoint so that we
can see where this huge cost is coming from? Checkpointing should be
limited to fsyncing to disk and marking WAL files as recyclable, but there
seems to be something more.

--
--Josh

Josh Berkus
Aglio Database Solutions
San Francisco