Re: swarm of processes in BIND state?

On Sat, May 28, 2016 at 08:04:43AM +0200, Pavel Stehule wrote:
> > > you should to install debug info - or compile with dubug symbols
> > Installed debug info, and the problem stopped.

OK. ot he problem back.

Ps looked like this:

USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND
postgres 9762 0.0 1.3 6816588 424512 ? S 05:49 0:09 /usr/lib/postgresql/9.3/bin/postgres -D /var/lib/postgresql/9.3/main -c config_file=/etc/postgresql/9.3/main/postgresql.conf
postgres 9763 1.6 20.3 6819980 6284520 ? Ss 05:49 11:49 \_ postgres: startup process recovering 0000000400007D61000000D2
postgres 10300 0.1 20.0 6827396 6190236 ? Ss 05:49 1:22 \_ postgres: checkpointer process
postgres 10301 0.1 19.9 6819736 6145144 ? Ss 05:49 0:55 \_ postgres: writer process
postgres 11111 0.0 0.0 104704 2076 ? Ss 05:49 0:11 \_ postgres: stats collector process
postgres 11148 0.7 0.0 6832848 4840 ? Ds 05:49 5:42 \_ postgres: wal receiver process streaming 7D61/D2CE4D90
postgres 43248 0.0 0.2 6823908 79072 ? Ss 06:41 0:00 \_ postgres: monitor dbname [local] idle
postgres 14802 4.0 18.5 6898688 5723652 ? Ss 09:57 19:54 \_ postgres: app_user dbname 10.1.15.177(44508) BIND
postgres 14804 2.9 16.3 6870980 5061020 ? Ss 09:57 14:25 \_ postgres: app_user dbname 10.1.15.177(44510) BIND
postgres 121044 6.2 16.8 6871756 5189808 ? Ss 13:19 17:52 \_ postgres: app_user dbname 10.1.14.77(58756) BIND
postgres 130506 5.0 15.6 6871536 4844704 ? Ss 13:38 13:22 \_ postgres: app_user dbname 10.1.15.177(47734) BIND
postgres 3767 2.8 13.6 6870876 4226928 ? Ss 13:45 7:20 \_ postgres: app_user dbname 10.1.14.77(59142) BIND
postgres 3768 3.7 16.0 6877232 4943916 ? Ss 13:45 9:35 \_ postgres: app_user dbname 10.1.14.77(59143) BIND
postgres 7168 0.0 0.0 6821248 8260 ? Ss 13:52 0:00 \_ postgres: pgbouncer dbname 10.1.16.158(50990) idle
postgres 7170 0.0 2.0 6863740 637368 ? Ss 13:52 0:01 \_ postgres: app_user dbname 10.1.16.158(50991) idle
postgres 23083 5.9 14.7 6872584 4552872 ? Ds 14:22 13:16 \_ postgres: app_user dbname 10.1.15.177(48363) BIND
postgres 70043 0.0 0.0 6821224 7548 ? Ss 15:52 0:00 \_ postgres: pgbouncer dbname 10.1.14.77(60980) idle
postgres 70719 0.0 0.0 6821248 7448 ? Ss 15:53 0:00 \_ postgres: pgbouncer dbname 10.1.15.177(49701) idle
postgres 71257 5.8 14.5 6870940 4479808 ? Ss 15:54 7:33 \_ postgres: app_user dbname 10.1.14.77(32783) BIND
postgres 77780 5.9 13.3 6871008 4119172 ? Ss 16:06 6:58 \_ postgres: app_user dbname 10.1.14.77(32951) BIND
postgres 84682 9.2 15.5 6877216 4806100 ? Ss 16:19 9:40 \_ postgres: app_user dbname 10.1.15.177(50074) BIND
postgres 100924 7.1 13.5 6870804 4176864 ? Ss 16:51 5:11 \_ postgres: app_user dbname 10.1.15.177(50537) BIND
postgres 118767 7.7 13.7 6870736 4251048 ? Ss 17:24 3:04 \_ postgres: app_user dbname 10.1.14.77(34090) BIND
postgres 118897 6.2 13.6 6870588 4219824 ? Ss 17:24 2:27 \_ postgres: app_user dbname 10.1.14.77(34094) BIND
postgres 118916 8.6 12.4 6869524 3852648 ? Ss 17:25 3:22 \_ postgres: app_user dbname 10.1.15.177(51026) BIND
postgres 118978 3.3 13.0 6868456 4033924 ? Ss 17:25 1:18 \_ postgres: app_user dbname 10.1.14.77(34100) BIND
postgres 119022 2.3 13.4 6829152 4148088 ? Ss 17:25 0:53 \_ postgres: app_user dbname 10.1.14.77(34101) BIND
postgres 119060 3.3 13.8 6869172 4265976 ? Ss 17:25 1:18 \_ postgres: app_user dbname 10.1.15.177(51034) BIND
postgres 119096 1.9 12.1 6828716 3754204 ? Ss 17:25 0:46 \_ postgres: app_user dbname 10.1.15.177(51037) BIND
postgres 119097 6.4 13.8 6870868 4263168 ? Ss 17:25 2:29 \_ postgres: app_user dbname 10.1.15.177(51038) BIND
postgres 119111 1.9 12.1 6826656 3763260 ? Ss 17:25 0:44 \_ postgres: app_user dbname 10.1.14.77(34105) BIND
postgres 119152 3.5 12.7 6869468 3921916 ? Ss 17:25 1:22 \_ postgres: app_user dbname 10.1.14.77(34107) BIND
postgres 119266 7.3 13.5 6868908 4193496 ? Ss 17:25 2:49 \_ postgres: app_user dbname 10.1.15.177(51041) BIND
postgres 119298 5.4 13.3 6870552 4107692 ? Ss 17:25 2:05 \_ postgres: app_user dbname 10.1.14.77(34110) BIND
postgres 119303 6.2 14.7 6870816 4553052 ? Ss 17:25 2:24 \_ postgres: app_user dbname 10.1.15.177(51043) BIND
postgres 119314 4.9 13.0 6869704 4024040 ? Ss 17:25 1:54 \_ postgres: app_user dbname 10.1.14.77(34113) BIND
postgres 119315 2.0 12.5 6869208 3881956 ? Ss 17:25 0:47 \_ postgres: app_user dbname 10.1.15.177(51045) BIND
postgres 119316 7.0 13.3 6870908 4136492 ? Ss 17:25 2:41 \_ postgres: app_user dbname 10.1.15.177(51046) BIND
postgres 119317 3.2 13.0 6870100 4018860 ? Ss 17:25 1:14 \_ postgres: app_user dbname 10.1.14.77(34114) BIND
postgres 119318 8.2 14.4 6870868 4457528 ? Ss 17:25 3:09 \_ postgres: app_user dbname 10.1.14.77(34115) BIND
postgres 119319 2.3 12.0 6827084 3724248 ? Ss 17:25 0:54 \_ postgres: app_user dbname 10.1.15.177(51047) BIND
postgres 119320 5.0 13.0 6870208 4037396 ? Ss 17:25 1:56 \_ postgres: app_user dbname 10.1.15.177(51048) BIND
postgres 119368 4.5 13.2 6868480 4085724 ? Ss 17:25 1:44 \_ postgres: app_user dbname 10.1.15.177(51050) BIND
postgres 119417 4.9 12.8 6869740 3981108 ? Ss 17:25 1:52 \_ postgres: app_user dbname 10.1.14.77(34117) BIND
postgres 119418 3.4 13.0 6869204 4027344 ? Ss 17:25 1:18 \_ postgres: app_user dbname 10.1.14.77(34118) BIND
postgres 119441 2.4 12.3 6828944 3826244 ? Ss 17:25 0:57 \_ postgres: app_user dbname 10.1.15.177(51051) BIND
postgres 119442 3.4 12.6 6869248 3896108 ? Ss 17:26 1:18 \_ postgres: app_user dbname 10.1.15.177(51052) BIND
postgres 119550 10.7 14.4 6870876 4448704 ? Ss 17:26 4:04 \_ postgres: app_user dbname 10.1.15.177(51062) BIND
postgres 119560 7.0 14.2 6870744 4402940 ? Ss 17:26 2:40 \_ postgres: app_user dbname 10.1.14.77(34125) BIND
postgres 119621 3.3 13.1 6870516 4052152 ? Ss 17:26 1:16 \_ postgres: app_user dbname 10.1.14.77(34128) BIND
postgres 119631 1.9 12.4 6827044 3858696 ? Ss 17:26 0:43 \_ postgres: app_user dbname 10.1.14.77(34129) BIND
postgres 130410 0.0 1.5 6824028 465040 ? Ss 17:47 0:00 \_ postgres: app_user dbname 10.1.15.177(51372) idle

I ran perf top and got:

Samples: 620K of event 'cpu-clock', Event count (approx.): 30018858044 [0/32]
23.77% postgres [.] s_lock
13.58% postgres [.] 0x000000000027668c
8.00% postgres [.] LWLockAcquire
6.45% [kernel] [k] _raw_spin_unlock_irqrestore
6.13% postgres [.] hash_search_with_hash_value
4.97% postgres [.] LWLockRelease
4.12% [kernel] [k] finish_task_switch
3.95% postgres [.] heap_page_prune_opt
1.33% postgres [.] RecoveryInProgress
1.19% [kernel] [k] copy_user_enhanced_fast_string
1.09% postgres [.] _bt_checkkeys
1.05% postgres [.] heap_hot_search_buffer
1.02% libc-2.19.so [.] 0x0000000000160b96
0.99% postgres [.] hash_any
...

Does that help us in any way?

Best regards,

depesz