Re: Server hitting 100% CPU usage, system comes to a crawl.

On Thu, Oct 27, 2011 at 02:09:51PM -0600, Brian Fehrle wrote:
- On 10/27/2011 01:48 PM, Scott Marlowe wrote:
- >On Thu, Oct 27, 2011 at 12:39 PM, Brian Fehrle
- ><brianf(at)consistentstate(dot)com> wrote:
- >>Looking at top, I see no SWAP usage, very little IOWait, and there are a
- >>large number of postmaster processes at 100% cpu usage (makes sense, at
- >>this
- >>point there are 150 or so queries currently executing on the database).
- >>
- >> Tasks: 713 total, 44 running, 668 sleeping, 0 stopped, 1 zombie
- >>Cpu(s): 4.4%us, 92.0%sy, 0.0%ni, 3.0%id, 0.0%wa, 0.0%hi, 0.3%si,
- >> 0.2%st
- >>Mem: 134217728k total, 131229972k used, 2987756k free, 462444k buffers
- >>Swap: 8388600k total, 296k used, 8388304k free, 119029580k cached
- >OK, a few points. 1: You've got a zombie process. Find out what's
- >causing that, it could be a trigger of some type for this behaviour.
- >2: You're 92% sys. That's bad. It means the OS is chewing up 92% of
- >your 32 cores doing something. what tasks are at the top of the list
- >in top?
- >
- Out of the top 50 processes in top, 48 of them are postmasters, one is
- syslog, and one is psql. Each of the postmasters have a high %CPU, the
- top ones being 80% and higher, the rest being anywhere between 30% -
- 60%. Would postmaster 'queries' that are running attribute to the sys
- CPU usage, or should they be under the 'us' CPU usage?

total spitball here but - I had something similar happen once and it
was syslog causing the problem.

Are you using regular vanilla syslog? or syslog-ng/rsyslog? my problem
was vanilla syslog. When I moved to -ng/rsyslog or logging to a file
my problem went away.

Dave