Re: Commit to primary with unavailable sync standby

On 19.12.2019 14:04, Andrey Borodin wrote:

> Hi!

Hi!

FYI, this topic was up recently in -hackers
https://www.postgresql.org/message-id/CAEET0ZHG5oFF7iEcbY6TZadh1mosLmfz1HLm311P9VOt7Z+jeg@mail.gmail.com

> I cannot figure out proper way to implement safe HA upsert. I will be very grateful if someone would help me.
>
> Imagine we have primary server after failover. It is network-partitioned. We are doing INSERT ON CONFLICT DO NOTHING; that eventually timed out.
>
> az1-grx88oegoy6mrv2i/db1 M > WITH new_doc AS (
> INSERT INTO t(
> pk,
> v,
> dt
> )
> VALUES
> (
> 5,
> 'text',
> now()
> )
> ON CONFLICT (pk) DO NOTHING
> RETURNING pk,
> v,
> dt)
> SELECT new_doc.pk from new_doc;
> ^CCancel request sent
> WARNING: 01000: canceling wait for synchronous replication due to user request
> DETAIL: The transaction has already committed locally, but might not have been replicated to the standby.
> LOCATION: SyncRepWaitForLSN, syncrep.c:264
> Time: 2173.770 ms (00:02.174)
>
> Here our driver decided that something goes wrong and we retry query.
>
> az1-grx88oegoy6mrv2i/db1 M > WITH new_doc AS (
> INSERT INTO t(
> pk,
> v,
> dt
> )
> VALUES
> (
> 5,
> 'text',
> now()
> )
> ON CONFLICT (pk) DO NOTHING
> RETURNING pk,
> v,
> dt)
> SELECT new_doc.pk from new_doc;
> pk
> ----
> (0 rows)
>
> Time: 4.785 ms
>
> Now we have split-brain, because we acknowledged that row to client.
> How can I fix this?
>
> There must be some obvious trick, but I cannot see it... Or maybe cancel of sync replication should be disallowed and termination should be treated as system failure?
>

I think the most appropriate way to handle such issues is to catch by
client driver such warnings (with message about local commit) and mark
the status of posted transaction as undetermined. If connection with
sync replica will come back then this transaction eventually commits but
after triggering of autofailover and *not replicating this commit to
replica* this commit aborts. Therefore client have to wait some time
(that exceeds the duration of autofailover) and check (logically based
on committed data) the status of commit.

The problem here is the locally committed data becomes visible to future
transactions (before autofailover) that violates the property of
consistent reading from master. IMO the more correct behavior for
PostgreSQL here is to ignore any cancel / termination queries when
backend is in status of waiting response from sync replicas.

However, there is another way to get locally applied commits via restart
of master after initial recovery. This case is described in doc
https://www.postgresql.org/docs/current/warm-standby.html#SYNCHRONOUS-REPLICATION-HA
. But here HA orchestrator agent can close access from external users
(via pg_hba.conf manipulations) until PostgreSQL instance synchronizes
its changes with all sync replicas as it's implemented in Stolon
https://github.com/sorintlab/stolon/blob/master/doc/syncrepl.md#handling-postgresql-sync-repl-limits-under-such-circumstances
.

Best regards,
Maksim Milyutin