pgsql: Wait between tablesync worker restarts

Wait between tablesync worker restarts

Before restarting a tablesync worker for the same relation, wait
wal_retrieve_retry_interval (currently 5s by default). This avoids
restarting failing workers in a tight loop.

We keep the last start times in a hash table last_start_times that is
separate from the table_states list, because that list is cleared out on
syscache invalidation, which happens whenever a table finishes syncing.
The hash table is kept until all tables have finished syncing.

A future project might be to unify these two and keep everything in one
data structure, but for now this is a less invasive change to accomplish
the original purpose.

For the test suite, set wal_retrieve_retry_interval to its minimum
value, to not increase the test suite run time.

Reviewed-by: Petr Jelinek <petr(dot)jelinek(at)2ndquadrant(dot)com>
Reported-by: Masahiko Sawada <sawada(dot)mshk(at)gmail(dot)com>

Branch
------
master

Details
-------
https://git.postgresql.org/pg/commitdiff/e3cf708016ca6045dc1cd5a0768cfecf17caf3d1

Modified Files
--------------
src/backend/replication/logical/tablesync.c | 58 ++++++++++++++++++++++++++---
src/test/subscription/t/004_sync.pl | 1 +
2 files changed, 53 insertions(+), 6 deletions(-)