Re: Issues with Quorum Commit

On 05.10.2010 22:11, Josh Berkus wrote:
> There's been a lot of discussion on synch rep lately which involves
> quorum commit. I need to raise some major design issues with quorum
> commit which I don't think that people have really considered, and may
> be sufficient to prevent it from being included in 9.1.

Thanks for bringing these up.

> A. Permanent Synchronization Failure
> ---------------------------------
> Quorum commit, like other forms of more-than-one-standby synch rep,
> offers the possibility that one or more standbys could end up
> irretrievably desyncronized with the master.
>
> 1. Quorum is 3 servers (out of 5) with mode "apply"
> 2. Standbys 2 and 4 receive and apply transaction # 20001.
> 3. Due to a network issue, no other standby applies #20001.
> 4. Accordingly, the master rolls back #20001 and cancels, either due to
> timeout or DBA cancel.

The master can not roll back or cancel the transaction. That's
completely infeasible, the WAL record has been written to local disk
already. The best it can do is halt and wait for enough standbys to
appear to fulfill the quorum. The client will hang waiting for the
COMMIT to finish, and the transaction will appear as in-progress to
other transactions.

There's subtle point here that I don't think has been discussed yet: If
the master is forcibly restarted at that point, with pg_ctl restart -m
immediate, strictly speaking the master should start up in the same
state, with the unlucky transaction still appearing as in-progress,
until the standby acknowledges.

> 5. #2 and #5 are now hopelessly out of synch with the master.

> B. Eventual Inconsistency
> -------------------------
> If we have a quorum commit, it's possible for any individual standby to
> be indefinitely ahead of any standby which is not needed by the quorum.
> This means that:
>
> -- There is no clear criteria for when a standby which is not needed for
> quorum should be considered no longer a synch standby, and
> -- Applications cannot make assumptions that synch rep promises some
> specific window of synchronicity, eliminating a lot of the value of
> quorum commit.

Yep.

> C. Performance
> --------------
> Doing quorum commit requires significant extra accounting on the
> master's part: it must keep track of how many standbys committed for
> each pending transaction (and remember there may be many at the same
> time).
>
> Doing so could involve significant response-time overhead added to the
> simple case where there is only one standby, as well as memory usage,
> and likely a lot of troubleshooting of the mechanism from us.

My gut feeling is that overhead will pale to insignificance compared to
the network and other overheads of actually getting the WAL to the
standby and processing the acknowledgments.

> D. Adding/Replacing Quorum Members
> ----------------------------------
> For Quorum commit to be really valuable, we need to be able to add new
> quorum members and remove dead ones *without stopping the master*. Per
> discussion about the startup issues with only one master, we have not
> worked out how to do this for synch rep standbys. It's reasonable to
> assume that this will be more complex for a quorum group than with a
> single synch standby.
>
> Consider the case, for example, where due to a network outage we have
> dropped below quorum. What is the strategy for getting the system
> running again by adding standbys?

You start a new one from the latest base backup and let it catch up?
Possibly modifying the config file in the master to let it know about
the new standby, if we go down that path. This part doesn't seem
particularly hard to me.

--
Heikki Linnakangas
EnterpriseDB http://www.enterprisedb.com