pgBackRest : "ERROR: [042]: unexpected EOF reading line from remote process on" and stale pgbackrest processes hanging

Hello List,

I had this error today with postgresql 10.6 and pgbackrest 2.07. I was running test backups and during a backup
The command given to the repo host (pg1-host=10.9.0.77) was :
pgbackrest(at)smadb2cs:~$ pgbackrest --stanza=dynacomtest backup
I noticed that the (two) ssh login sessions to the repo host just froze. No disconnections, just froze. So some network problem surely occurred.
On the repo host pgbackrest log I got :
2019-01-16 12:20:38.372 P00  ERROR: [042]: unexpected EOF reading line from remote process on '10.9.0.77'
on the pg host (10.9.0.77) I noticed that the pgbackrest backup processes had died, the only ones running was spawn from the archive-push , however on the repo host the pgbackrest processes were
alive, when in fact the backup was stalled.
On the repo I got :
root(at)smadb2cs:/home/pgbackrest# netstat -apn | grep 10.9.0.77
tcp        0     88 10.9.1.22:43837 10.9.0.77:22            ESTABLISHED 11918/ssh
tcp        0     88 10.9.1.22:43838 10.9.0.77:22            ESTABLISHED 11920/ssh
tcp        0     88 10.9.1.22:43839 10.9.0.77:22            ESTABLISHED 11919/ssh
tcp        0     88 10.9.1.22:43840 10.9.0.77:22            ESTABLISHED 11922/ssh
root(at)smadb2cs:/home/pgbackrest# netstat -anp --timers | grep 0.77
tcp        0     88 10.9.1.22:43837 10.9.0.77:22            ESTABLISHED 11918/ssh        on (65.50/12/0)
tcp        0     88 10.9.1.22:43838 10.9.0.77:22            ESTABLISHED 11920/ssh        on (69.60/12/0)
tcp        0     88 10.9.1.22:43839 10.9.0.77:22            ESTABLISHED 11919/ssh        on (65.50/12/0)
tcp        0     88 10.9.1.22:43840 10.9.0.77:22            ESTABLISHED 11922/ssh        on (61.40/12/0)

In order to start a new backup I had to kill the 4 processes.

Is this behavior normal ? Would the 4 processes be killed due to KEEPALIVE settings eventually ?

--
Achilleas Mantzios
IT DEV Lead
IT DEPT
Dynacom Tankers Mgmt