Re: odd iostat graph

On 9/11/09 9:58 AM, "Alan McKay" <alan(dot)mckay(at)gmail(dot)com> wrote:

> Hey folks,
>
> Earlier in the week I wrote a Munin plugin that takes the "await" and
> "average queue length" fields from "iostat -x" and graphs them.
>
> This seems rather odd to me :
>
> http://picasaweb.google.ca/alan.mckay/Work#5380253477470243954
>
> That is Qlen. And await looks similar
>
> http://picasaweb.google.ca/alan.mckay/Work#5380254090296723426
>
> This is on an IBM 3650 with the 2 main "internal" drives set up in a
> mirrored config, and sdb are the 6 other drives set up in a RAID5 with
> a global hot spare. (4 drives in array + 1 to make it RAID5 + global
> hot spare)
>
> We aren't seeing any performance problems on this per-se. But that
> just seems like a really odd graph to me. Can anyone explain it? In
> particular, how regular it is?

My guess is this is checkpoint related.
Find out when your checkpoints are happening. The drops are most likely due
to the sync() on all outstanding writes at the end of each checkpoint. The
rise is probably small writes not yet on disk in the OS bufffer cache. If
this is due to checkpoints, I would expect a burst of write volume to disk
at the same time of the drop.

You can change your logging settings to output the time of each checkpoint
and some stats about them.

>
> cheers,
> -Alan
>
> --
> ³Don't eat anything you've ever seen advertised on TV²
> - Michael Pollan, author of "In Defense of Food"
>
> --
> Sent via pgsql-performance mailing list (pgsql-performance(at)postgresql(dot)org)
> To make changes to your subscription:
> http://www.postgresql.org/mailpref/pgsql-performance
>