Issues with Quorum Commit

All,

There's been a lot of discussion on synch rep lately which involves
quorum commit. I need to raise some major design issues with quorum
commit which I don't think that people have really considered, and may
be sufficient to prevent it from being included in 9.1.

A. Permanent Synchronization Failure
---------------------------------
Quorum commit, like other forms of more-than-one-standby synch rep,
offers the possibility that one or more standbys could end up
irretrievably desyncronized with the master.

1. Quorum is 3 servers (out of 5) with mode "apply"
2. Standbys 2 and 4 receive and apply transaction # 20001.
3. Due to a network issue, no other standby applies #20001.
4. Accordingly, the master rolls back #20001 and cancels, either due to
timeout or DBA cancel.
5. #2 and #5 are now hopelessly out of synch with the master.

B. Eventual Inconsistency
-------------------------
If we have a quorum commit, it's possible for any individual standby to
be indefinitely ahead of any standby which is not needed by the quorum.
This means that:

-- There is no clear criteria for when a standby which is not needed for
quorum should be considered no longer a synch standby, and
-- Applications cannot make assumptions that synch rep promises some
specific window of synchronicity, eliminating a lot of the value of
quorum commit.

C. Performance
--------------
Doing quorum commit requires significant extra accounting on the
master's part: it must keep track of how many standbys committed for
each pending transaction (and remember there may be many at the same
time).

Doing so could involve significant response-time overhead added to the
simple case where there is only one standby, as well as memory usage,
and likely a lot of troubleshooting of the mechanism from us.

D. Adding/Replacing Quorum Members
----------------------------------
For Quorum commit to be really valuable, we need to be able to add new
quorum members and remove dead ones *without stopping the master*. Per
discussion about the startup issues with only one master, we have not
worked out how to do this for synch rep standbys. It's reasonable to
assume that this will be more complex for a quorum group than with a
single synch standby.

Consider the case, for example, where due to a network outage we have
dropped below quorum. What is the strategy for getting the system
running again by adding standbys?

All of the problems above are resolvable. Some of the CAP databases
have probably resolved them, as well as some older telecom databases.
However, all of them will require significant work, and even more
significant debugging, from the project.

I would like to see Quorum Commit, in part because I think it would help
push PostgreSQL further into cloud frameworks. However, I'm worried
that if we make quorum commit a requirement of synch rep, we will not
have synch rep in 9.1.

--
-- Josh Berkus
PostgreSQL Experts Inc.
http://www.pgexperts.com