Hi,
I think I solved it. This is so ridiculous!
The primary and standby databases are 200km apart and the path between them
has a bandwidth of only 100 Mb/s. We had bursts of writes that created
huge amounts of replication traffic. In addition to streaming
replication, we're still copying WAL files over for safety (and also
because this is 9.1...)
We were hitting our bandwidth limit.
By adding the --bwlimit parameter to the rsync command that copies the WAL
files, we can limit the bandwidth usage during the bursts while still
keeping it high enough for the WAL-file archiving to keep up on average.
Keeping an eye on it for 24h before I completely declare it solved.
Regards,
Dianne.