From: | Achilleas Mantzios <achill(at)matrix(dot)gatewaynet(dot)com> |
---|---|
To: | Stephen Frost <sfrost(at)snowman(dot)net> |
Cc: | pgsql-admin(at)lists(dot)postgresql(dot)org |
Subject: | Re: pgBackRest : "ERROR: [042]: unexpected EOF reading line from remote process on" and stale pgbackrest processes hanging |
Date: | 2019-01-17 05:20:27 |
Message-ID: | 8d6ac7ad-8afe-7802-34b8-0e2f368f6612@matrix.gatewaynet.com |
Views: | Raw Message | Whole Thread | Download mbox | Resend email |
Thread: | |
Lists: | pgsql-admin |
Hello Stephen
On 16/1/19 10:27 μ.μ., Stephen Frost wrote:
> Greetings,
>
> * Achilleas Mantzios (achill(at)matrix(dot)gatewaynet(dot)com) wrote:
>> I had this error today with postgresql 10.6 and pgbackrest 2.07. I was running test backups and during a backup
>> The command given to the repo host (pg1-host=10.9.0.77) was :
>> pgbackrest(at)smadb2cs:~$ pgbackrest --stanza=dynacomtest backup
>> I noticed that the (two) ssh login sessions to the repo host just froze. No disconnections, just froze. So some network problem surely occurred.
>> On the repo host pgbackrest log I got :
>> 2019-01-16 12:20:38.372 P00 ERROR: [042]: unexpected EOF reading line from remote process on '10.9.0.77'
>> on the pg host (10.9.0.77) I noticed that the pgbackrest backup processes
>> had died, the only ones running was spawn from the archive-push , however on
>> the repo host the pgbackrest processes were alive, when in fact the backup
>> was stalled.
>> On the repo I got :
>> root(at)smadb2cs:/home/pgbackrest# netstat -apn | grep 10.9.0.77
>> tcp 0 88 10.9.1.22:43837 10.9.0.77:22 ESTABLISHED 11918/ssh
>> tcp 0 88 10.9.1.22:43838 10.9.0.77:22 ESTABLISHED 11920/ssh
>> tcp 0 88 10.9.1.22:43839 10.9.0.77:22 ESTABLISHED 11919/ssh
>> tcp 0 88 10.9.1.22:43840 10.9.0.77:22 ESTABLISHED 11922/ssh
>> root(at)smadb2cs:/home/pgbackrest# netstat -anp --timers | grep 0.77
>> tcp 0 88 10.9.1.22:43837 10.9.0.77:22 ESTABLISHED 11918/ssh on (65.50/12/0)
>> tcp 0 88 10.9.1.22:43838 10.9.0.77:22 ESTABLISHED 11920/ssh on (69.60/12/0)
>> tcp 0 88 10.9.1.22:43839 10.9.0.77:22 ESTABLISHED 11919/ssh on (65.50/12/0)
>> tcp 0 88 10.9.1.22:43840 10.9.0.77:22 ESTABLISHED 11922/ssh on (61.40/12/0)
> This looks like the kernel thinks the connections are still alive.
>
>> In order to start a new backup I had to kill the 4 processes.
>>
>> Is this behavior normal ? Would the 4 processes be killed due to KEEPALIVE settings eventually ?
> Yes, they almost certainly would have timed out eventually- but how
> long were you already waiting..? The timeouts in pgbackrest are not
> terribly long, in general.
Looking at the log I waited 9 minutes, before manually killing.
>
> Thanks!
>
> Stephen
From | Date | Subject | |
---|---|---|---|
Next Message | David Steele | 2019-01-17 06:50:27 | Re: pgBackRest : "ERROR: [042]: unexpected EOF reading line from remote process on" and stale pgbackrest processes hanging |
Previous Message | chris | 2019-01-17 00:36:05 | Re:how to store data file in Postgres |