Postgresql 11: terminating walsender process due to replication timeout

From: Abhishek Bhola <abhishek(dot)bhola(at)japannext(dot)co(dot)jp>
To: pgsql-general(at)postgresql(dot)org
Subject: Postgresql 11: terminating walsender process due to replication timeout
Date: 2021-09-09 05:52:25
Message-ID: CAEDsCzhBtkNDLM46_fo_HirFYE2Mb3ucbZrYqG59ocWqWy7-xA@mail.gmail.com
Views: Raw Message | Whole Thread | Download mbox | Resend email
Thread:
Lists: pgsql-general

I have found some questions about the same error, but didn't find any of
them answering my problem.

The setup is that I have two Postgres11 clusters (A and B) and they are
making use of publication and subscription features to copy data from A to
B.

A (source DB- publication) --------------> B (target DB - subscription)

This works fine, but often (not always) when the data volume being inserted
on a table in node A increases, it gives the following error.

"terminating walsender process due to replication timeout"

The data volume at the moment being entered is about 30K rows per second
continuously for hours through COPY command.

Earlier the wal_sender_timeout was set to 5 sec and I would see this error
much often. I then increased it to 1 min and the frequency of this error
reduced. But I don't want to keep increasing it without understanding what
is causing it. I looked at the code of walsender.c and know the exact lines
where it's coming from.

But I am still not clear which parameter is making the sender assume that
the receiver node is inactive and therefore it should stop the wal_sender.

Can anyone please suggest what changes I should make to remove this error?

sourcedb=# show wal_sender_timeout;
wal_sender_timeout--------------------
1min
(1 row)

sourcedb=# select * from pg_replication_slots;
slot_name | plugin | slot_type | datoid |
database | temporary | active | active_pid | xmin | catalog_xmin |
restart_lsn | confirmed_flush_lsn
------------------------------------+----------+-----------+--------+----------+-----------+--------+------------+------+--------------+----------------+---------------------
sub_target_DB | pgoutput | logical | 16501 |
sourcedb | f | t | 68229 | | 98839088 |
116D0/C36886F8 | 116D0/C3E5D370

targetdb=# show wal_receiver_timeout;
wal_receiver_timeout----------------------
1min
(1 row)

targetdb=# show wal_retrieve_retry_interval ;
wal_retrieve_retry_interval-----------------------------
5s
(1 row)

targetdb=# show wal_receiver_status_interval;
wal_receiver_status_interval------------------------------
2s
(1 row)

targetdb=# select * from pg_stat_subscription;
subid | subname | pid | relid |
received_lsn | last_msg_send_time |
last_msg_receipt_time | latest_end_lsn | l
atest_end_time------------+------------------------------------+-------+-------+----------------+-------------------------------+-------------------------------+----------------+-------------------------------
2378695757 | sub_target_DB | 62371 | |
116D1/2BA8F170 | 2021-08-20 09:05:15.398423+09 | 2021-08-20
09:05:15.398471+09 | 116D1/2BA8F170 | 2021-08-20 09:05:15.398423+09

Increased the wal_sender_timeout to 5 mins and the error started
appearing more frequently instead. Not only that, it even killed the
active subscription and stopped replicating data. Had to restart it.
So clearly, just increasing the wal_sender_timeout hasn't helped.

--
_This correspondence (including any attachments) is for the intended
recipient(s) only. It may contain confidential or privileged information or
both. No confidentiality or privilege is waived or lost by any
mis-transmission. If you receive this correspondence by mistake, please
contact the sender immediately, delete this correspondence (and all
attachments) and destroy any hard copies. You must not use, disclose, copy,
distribute or rely on any part of this correspondence (including any
attachments) if you are not the intended
recipient(s).本メッセージに記載および添付されている情報(以下、総称して「本情報」といいます。)は、本来の受信者による使用のみを意図しています。誤送信等により本情報を取得された場合でも、本情報に係る秘密、または法律上の秘匿特権が失われるものではありません。本電子メールを受取られた方が、本来の受信者ではない場合には、本情報及びそのコピーすべてを削除・破棄し、本電子メールが誤って届いた旨を発信者宛てにご通知下さいますようお願いします。本情報の閲覧、発信または本情報に基づくいかなる行為も明確に禁止されていることをご了承ください。_

Responses

Browse pgsql-general by date

  From Date Subject
Next Message Avi Weinberg 2021-09-09 05:53:58 Subscriber to Get Only Some of The Tables From Publisher
Previous Message Celia McInnis 2021-09-09 03:55:20 Re: spiexceptions.UndefinedFile: could not open file "base/16482/681830": No such file or directory