Re: processor running queue - general rule of thumb?

On Fri, Jun 19, 2009 at 9:59 AM, Alan McKay<alan(dot)mckay(at)gmail(dot)com> wrote:
> Hey folks,
>
> I'm new to all this stuff, and am sitting here with kSar looking at
> some graphed results of some load tests we did, trying to figure
> things out :-)
>
> We got some unsatisfactory results in stressing our system, and now I
> have to divine where the bottleneck is.
>
> We did 4 tests, upping the load each time.   The 3rd and 4th ones have
> all 8 cores pegged at about 95%.  Yikes!
>
> In the first test the processor running queue spikes at 7 and maybe
> averages 4 or 5
>
> In the last test it spikes at 33 with an average maybe 25.
>
> Looks to me like it could be a CPU bottleneck.  But I'm new at this :-)
>
> Is there a general rule of thumb "if queue is longer than X, it is
> likely a bottleneck?"
>
> In reading an IBM Redbook on Linux performance, I also see this :
> "High numbers of context switches in connection with a large number of
> interrupts can signal driver or application issues."
>
> On my first test where the CPU is not pegged, context switching goes
> from about 3700 to about 4900, maybe averaging 4100

That's not too bad. If you see them in the 30k to 150k range, then
worry about it.

> On the pegged test, the values are maybe 10% higher than that, maybe 15%.

That's especially good news. Normally when you've got a problem, it
will increase in a geometric (or worse) way.

> It is an IBM 3550 with 8 cores, 2660.134 MHz (from dmesg), 32Gigs RAM

Like the other poster said, we likely don't have enough to tell you
what's going on, but from what you've said here it sounds like you're
mostly just CPU bound. Assuming you're reading the output of vmstat
and top and other tools like that.