BUG #9635: Wal sender process is using 100% CPU

The following bug has been logged on the website:

Bug reference: 9635
Logged by: Jamie Koceniak
Email address: jkoceniak(at)mediamath(dot)com
PostgreSQL version: 9.1.9
Operating system: x86_64-unknown-linux-gnu (Debian 4.7.2-5) 64-bit
Description:

Periodically throughout the day, we keep seeing the wal sender process
utilize 100% of the CPU. We began noticing this after we added 2 new slave
servers, going from 2 to 4 slaves. See top results and I also included our
wal settings. Thanks!

top - 05:03:18 up 174 days, 4:51, 2 users, load average: 5.57, 4.75,
3.16
Tasks: 387 total, 8 running, 379 sleeping, 0 stopped, 0 zombie
%Cpu(s): 29.3 us, 4.7 sy, 0.0 ni, 65.3 id, 0.4 wa, 0.0 hi, 0.4 si, 0.0
st
MiB Mem: 290797 total, 218532 used, 72264 free, 311 buffers
MiB Swap: 7812 total, 1 used, 7811 free, 206978 cached

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND

30244 postgres 20 0 8497m 5608 2820 R 100 0.0 1:44.72 postgres: wal
sen
14447 postgres 20 0 8497m 5596 2816 R 100 0.0 3:11.27 postgres: wal
sen
16075 postgres 20 0 8497m 5600 2820 R 100 0.0 3:32.32 postgres: wal
sen
8177 postgres 20 0 8497m 5360 2820 S 36 0.0 0:03.35 postgres: wal
sen
4920 postgres 20 0 9647m 9.3g 8.1g S 3 3.3 1097:40 postgres:
writer
4923 postgres 20 0 68872 2072 788 S 3 0.0 511:01.76 postgres:
archive
4921 postgres 20 0 8496m 18m 17m S 2 0.0 593:36.38 postgres: wal
wri
7853 root 20 0 23432 1836 1176 R 1 0.0 0:00.44 top

4916 postgres 20 0 8492m 229m 228m S 0 0.1 598:44.57
/usr/lib/postgres

Current Wal settings:

name | setting |
------------------------------+-------------+
max_wal_senders | 10 |
wal_block_size | 8192 |
wal_buffers | 2048 |
wal_keep_segments | 5000 |
wal_level | hot_standby |
wal_receiver_status_interval | 10 |
wal_segment_size | 2048 |
wal_sender_delay | 1000 |
wal_sync_method | fdatasync |
wal_writer_delay | 200 |