Re: High SYS CPU - need advise

Some additional observation and food for thoughts. Our app uses connection
caching (Apache::DBI). By disabling Apache::DBI and forcing
client re-connection for every (http) request processed I eliminated the
stall. The user cpu usage jumped (mostly cause prepared sql queries are no
longer available, and some additional overhead on re-connection), but no
single case of high-sys-cpu stall.

I can not completely rule out the possibility of some left-overs
(unfinished transaction?) remain after serving http request, which, in the
absence of connection caching, are discarded for sure....

-- Vlad

On Mon, Nov 19, 2012 at 11:19 AM, Merlin Moncure <mmoncure(at)gmail(dot)com> wrote:

>
> yeah. interesting -- contention was much higher this time and that
> changes things. strange how it was missed earlier.
>
> you're getting bounced around a lot in lwlock especially
> (unfortunately we don't know which one). I'm going to hazard another
> guess: maybe the trigger here is when the number of contending
> backends exceeds some critical number (probably based on the number of
> cores) you see a quick cpu spike (causing more backends to lock and
> pile up) as cache line bouncing sets in. That spike doesn't last
> long, because the spinlocks quickly accumulate delay counts then punt
> to the scheduler which is unable to cope. The exact reason why this
> is happening to you in exactly this way (I've never seen it) is
> unclear. Also the line between symptom and cause is difficult to
> draw.
>
> unfortunately, in your case spinlock re-scheduling isn't helping. log
> entries like this one:
> 18764 [2012-11-19 10:43:50.124 CST] LOG: JJ spin delay from file
> sinvaladt.c line 512 delay 212, pointer 0x7f514959a394 at character 29
>
> are suggesting major problems. you're dangerously close to a stuck
> spinlock which is lights out for the database.
>
> merlin
>