connections not getting closed on a replica

Hello,

I'm trying to figure out why we had a build up of connections on our
streaming replica. We're running postgres 9.3.5 on the master and 9.3.10 on
the replica, linux 3.2.0 on both, disks in raid10. Here are some graphs to
illustrate what happened:

http://dd-pastebin.s3.amazonaws.com/carlo/pg-connections-issue.png

There were 3 different instances, as shown by the spikes in server
connections graph. Each incident, the replica was still serving some
requests but much slower, and connections kept building up.
Pg_cancel_backend(), pg_terminate_backend() and the server timeout of 5s
weren't closing them. Eventually we had to restart the database to get back
to normal which isn't ideal.

The second graph shows the rows updated and hot updated on the master
(n_tup_upd and n_tup_hot_upd from pg_stat_user_tables) on a large,
frequently updated table (27M rows, 20GB). It looks like hot updates start
going down right before connections run away. Going through the code, I
found this comment:
https://github.com/postgres/postgres/blob/REL9_3_5/src/backend/access/heap/heapam.c#L3031-L3041

Which from my understanding means that a hot update is a weaker lock, but
if column is being "detoasted" an update will use the stronger lock
instead. I'm not sure what detoasted means, but I do know that table has a
toast table as it's using an hstore column. We can store pretty large
values in the hstore column, so I wouldn't be surprised if that was causing
the issue.

The third graph shows toast block cache hits o (toast_blks_hit from
pg_statio_user_tables) and shows that the hits on the replica (purple) took
a nose dive after the first incident and pretty much never recovered since
then. Does that mean the toast table is too large to fit in cache?

The fourth graph shows the rate of dead row creation on the master, with
the large blue area being the table in question. I'm assuming the dead rows
are from updates on the hstore, but not certain.

The fifth graph shows the disk write throughput on the replica, which I'm
assuming is from applying the WAL updates on that large table? Write
throughput was higher than normal, but not maxing out the capacity of the
disk. Write latency was minimal.

The 6th graph is the replica lag, which only fell behind during the first
incident and not the others.

My question is why was the postgres replica unable to close connections
during these incidents? And why were reads on the replica so slow at the
time? Are there locks that are locking out the readers when there's lots of
update churn?

Carlo Cabanilla
Datadog