Re: BUG #7534: walreceiver takes long time to detect n/w breakdown

On 04.10.2012 13:12, Amit kapila wrote:
> Following changes are done to support replication timeout in sender as well as receiver:
>
> 1. One new configuration parameter wal_receiver_timeout is added to detect timeout at receiver task.
> 2. Existing parameter replication_timeout is renamed to wal_sender_timeout.

Ok. The other option would be to have just one GUC, I'm open to
bikeshedding on this one. On one hand, there's no reason the timeouts
have to the same, so it would be nice to have separate settings, but on
the other hand, I can't imagine a case where a single setting wouldn't
work just as well.

> 3. Now PrimaryKeepaliveMessage structure is modified to add one more field to indicate whether keep-alive is of type 'r' (i.e.
> reply) or 'h' (i.e. heart-beat).
> 4. Now the keep-alive message from sender will be sent to standby if it was idle for more than or equal to half of wal_sender_timeout.
> In this case it will send keep-alive of type 'h'.
> 5. Once the standby receiver a keep-alive, it needs to send an immediate reply to primary to indicate connection is alive.
> 6. Now Reply message to send wal offset and Feedback message to send oldest transaction are merged into single Reply message.
> So now the structure StandbyReplyMessage is changed to add two more fields as xmin and epoch. Also StandbyHSFeedbackMessage
> structure is changed to remove xmin and epoch fields (as these are moved to StandbyReplyMessage).
> 7. Because of changes as in step-6, once receiver task receives some data from primary then it will only send Reply Message.

Oh I see. That's not what I meant by combining the keep-alive and hs
feedback messages, I imagined that the hearbeats would *also* use the
same message type. Ie. there would be only a single message type from
standby to primary, used for:

1. updating the receive/apply pointer
2. HS feedback
3. for pinging the server when wal_receiver_timeout is approaching
4. to reply to to pings from the server.

Since we didn't quite achieve that, it seems best leave out this merging
of reply and HS feedback message types, to keep the patch small. We
might still want to do that, but better do that as a separate patch.

> 8. Same Reply message is sent in step-5 and step-7 but incase of step-5, then reply is sent immediately but incase of step-7, reply is sent
> if wal_receiver_status_interval has lapsed (this part is same as earlier).
> 9. Similar to sender, if receiver finds itself idle for more than or equal to half of configured wal_receiver_timeout, then it will send the
> hot-standby heartbeat. This heart-beat has been modified to send only sendTime.
> 10. Once sender task receiver heart-beat message from standby then it sends back the reply immediately. In this keep-alive message is
> sent of type 'r'.
> 11. If even after wal_sender_timeout no message received from standby then it will be considered as network break at sender task.
> 12. If even after wal_receiver_timeout no message received from primary then it will be considered as network break at receiver task.

Attached is an updated patch. I reverted the merging of message types
and fixed a bunch of cosmetic issues. There was one bug: in the main
loop of walreceiver, you send the "ping" message on every wakeup after
enough time has passed since last reception. That means that if the
server doesn't reply promptly, you send a new ping message every 100 ms
(NAPTIME_PER_CYCLE), until it gets a reply. Walsender had the same
issue, but it was not quite as sever there because the naptime was
longer. Fixed that.

How does this look now?

- Heikki