100% cpu usage on some postmaster processes kill the complete database

From: Piotr Ciechomski <p(dot)ciechomski(at)focustelecom(dot)pl>
To: pgsql-general(at)postgresql(dot)org
Subject: 100% cpu usage on some postmaster processes kill the complete database
Date: 2012-05-18 11:31:56
Message-ID: CADzsWg9PEhj-xT7AWzWc6Jabom0+joxVgKZx71PohFB0i-bd3g@mail.gmail.com
Views: Raw Message | Whole Thread | Download mbox | Resend email
Thread:
Lists: pgsql-general

CentOS 6.0 - 2.6.32-220.13.1.el6.x86_64
PostgreSQL version: 8.4.11

Primarily checked on PG 8.4.9 (same OS), problem also occurs. Few times a
day I get a situation where PostgreSQL stops running for 1-2 minutes. CPU is
running 99% in systime. IO is OK, only interrupts are extremely high (over
100k up to 400K). System operates on 2 x Xeon 10 Core, 128 GB RAM, raid 10.
Does anyone have any idea?

monitoring from atop

ATOP - 2012/05/18 09:26:16 ------
10s elapsed
PRC | sys 6m29s | user 8m56s | #proc 1155 | #tslpi 820 |
#tslpu 7 | #zombie 2 | #exit 2 |
CPU | sys 3995% | user 3% | irq 0% | idle 4% | wait
0% | curf 2.00GHz | curscal ?% |
CPL | avg1 83.19 | avg5 21.01 | avg15 9.11 | csw 3036 | intr
448760 | | numcpu 40 |
MEM | tot 126.2G | free 2.6G | cache 117.6G | dirty 17.6M | buff
277.1M | slab 1.7G | |
SWP | tot 0.0M | free 0.0M | | |
| vmcom 41.5G | vmlim 63.1G |
PAG | scan 8333 | stall 0 | | | swin
0 | | swout 0 |
DSK | sda | busy 1% | read 9 | write 2 | MBr/s
0.01 | MBw/s 0.00 | avio 9.09 ms |
NET | transport | tcpi 127 | tcpo 99 | udpi 0 | udpo
3 | tcpao 0 | tcppo 36 |
NET | network | ipi 138 | ipo 103 | ipfrw 0 | deliv
130 | icmpi 0 | icmpo 0 |
NET | eth0 0% | pcki 137 | pcko 109 | si 10 Kbps | so
8 Kbps | erri 0 | erro 0 |
NET | eth1 0% | pcki 9 | pcko 0 | si 0 Kbps | so
0 Kbps | erri 0 | erro 0 |
NET | lo ---- | pcki 3 | pcko 3 | si 1 Kbps | so
1 Kbps | erri 0 | erro 0 |

PID RUID EUID THR SYSCPU USRCPU VGROW RGROW RDDSK
WRDSK ST EXC S CPUNR CPU CMD 1/34
64574 postgres - 0 9.44s 8m49s 0K 0K -
- NE 0 E - 100% <postmaster>
23 root root 1 5.22s 0.00s 0K 0K 0K
0K -- - S 5 50% migration/5
26885 postgres postgres 1 4.93s 0.00s 0K 4096K 0K
0K -- - R 25 47% postmaster
71 root

Browse pgsql-general by date

  From Date Subject
Next Message Seref Arikan 2012-05-18 11:57:00 Re: Re: Is EXISTS the most efficient approach for PostgreSql to check for existence of nodes in a tree?
Previous Message Vincenzo Romano 2012-05-18 11:20:19 How to check for server availability? [v9.3.1, Linux]