Re: Listen / Notify - what to do when the queue is full

Hi,

On Mon, Dec 7, 2009 at 5:38 AM, Greg Smith <greg(at)2ndquadrant(dot)com> wrote:
> JI'm going to mark this one "returned with feedback", and I
> do hope that work continues on this patch so it's early in the queue for the
> final CommitFest for this version.  It seems like it just needs a bit more
> time to let the design mature and get the kinks worked out and it could turn
> into a useful feature--I know I've wanted NOTIFY with a payload for a years.

I am perfectly fine with postponing the patch to the next commitfest. To get
some more feedback and to allow everyone to play with it, I am attaching the
latest version of the patch.

What has changed:

Transactional processing is now hopefully correct:

Examples:

Backend 1: Backend 2:

transaction starts
NOTIFY foo;
commit starts
transaction starts
LISTEN foo;
commit starts
commit to clog
commit to clog

=> Backend 2 will receive Backend 1's notification.

Backend 1: Backend 2:

transaction starts
NOTIFY foo;
commit starts
transaction starts
UNLISTEN foo;
commit starts
commit to clog
commit to clog

=> Backend 2 will not receive Backend 1's notification.

This is done by introducing an additional "AsyncCommitOrderLock". It is grabbed
exclusively from transactions that execute LISTEN / UNLISTEN and in shared mode
for transactions that executed NOTIFY only. LISTEN/UNLISTEN transactions then
register the XIDs of the NOTIFYing transactions that are about to commit
at the same time in order to later find out which notifications are visible and
which ones are not.

If the queue is full, any other transaction that is trying to place a
notification to the queue is rolled back! This is basically a consequence of
the former. There are two warnings that will show up in the log once the queue
is more than 50% full and another one if it is more than 75% full. The biggest
threat to run into a full queue are probably backends that are LISTENing and
are idle in transaction.

I have added a function pg_listening() which just contains the names of the
channels that a backend is listening to.

I especially invite people who know more about the transactional stuff than I
do to take a close look at what I have done regarding notification visibility.

One open question regarding the payload is if we need to limit it to ASCII to
not risk conversion issues between different backend character sets?

The second open issue is what we should do regarding 2PC. These options have
been brought up so far:

1) allow NOTIFY in 2PC but it can happen that the transaction needs to be
rolled back if the queue is full
2) disallow NOTIFY in 2PC alltogether
3) put notifications to the queue on PREPARE TRANSACTION and make backends not
advance their pointers further than those notifications but wait for the
2PC transaction to commit. 2PC transactions would never fail but you
effectively stop the notification system until the 2PC transaction commits.

Comments?

Best regards,
Joachim