processor running queue - general rule of thumb?

Hey folks,

I'm new to all this stuff, and am sitting here with kSar looking at
some graphed results of some load tests we did, trying to figure
things out :-)

We got some unsatisfactory results in stressing our system, and now I
have to divine where the bottleneck is.

We did 4 tests, upping the load each time. The 3rd and 4th ones have
all 8 cores pegged at about 95%. Yikes!

In the first test the processor running queue spikes at 7 and maybe
averages 4 or 5

In the last test it spikes at 33 with an average maybe 25.

Looks to me like it could be a CPU bottleneck. But I'm new at this :-)

Is there a general rule of thumb "if queue is longer than X, it is
likely a bottleneck?"

In reading an IBM Redbook on Linux performance, I also see this :
"High numbers of context switches in connection with a large number of
interrupts can signal driver or application issues."

On my first test where the CPU is not pegged, context switching goes
from about 3700 to about 4900, maybe averaging 4100

On the pegged test, the values are maybe 10% higher than that, maybe 15%.

It is an IBM 3550 with 8 cores, 2660.134 MHz (from dmesg), 32Gigs RAM

thanks,
-Alan

--
“Don't eat anything you've ever seen advertised on TV”
- Michael Pollan, author of "In Defense of Food"