Re: Listen / Notify - what to do when the queue is full

On Wed, Dec 9, 2009 at 5:43 AM, Joachim Wieland <joe(at)mcknight(dot)de> wrote:
> Hi,
>
> On Mon, Dec 7, 2009 at 5:38 AM, Greg Smith <greg(at)2ndquadrant(dot)com> wrote:
>> JI'm going to mark this one "returned with feedback", and I
>> do hope that work continues on this patch so it's early in the queue for the
>> final CommitFest for this version.  It seems like it just needs a bit more
>> time to let the design mature and get the kinks worked out and it could turn
>> into a useful feature--I know I've wanted NOTIFY with a payload for a years.
>
> I am perfectly fine with postponing the patch to the next commitfest. To get
> some more feedback and to allow everyone to play with it, I am attaching the
> latest version of the patch.
>
> What has changed:
>
> Transactional processing is now hopefully correct:
>
> Examples:
>
> Backend 1:                    Backend 2:
>
> transaction starts
> NOTIFY foo;
> commit starts
>                              transaction starts
>                              LISTEN foo;
>                              commit starts
>                              commit to clog
> commit to clog
>
> => Backend 2 will receive Backend 1's notification.
>
>
> Backend 1:                    Backend 2:
>
> transaction starts
> NOTIFY foo;
> commit starts
>                              transaction starts
>                              UNLISTEN foo;
>                              commit starts
>                              commit to clog
> commit to clog
>
> => Backend 2 will not receive Backend 1's notification.
>
> This is done by introducing an additional "AsyncCommitOrderLock". It is grabbed
> exclusively from transactions that execute LISTEN / UNLISTEN and in shared mode
> for transactions that executed NOTIFY only. LISTEN/UNLISTEN transactions then
> register the XIDs of the NOTIFYing transactions that are about to commit
> at the same time in order to later find out which notifications are visible and
> which ones are not.
>
> If the queue is full, any other transaction that is trying to place a
> notification to the queue is rolled back! This is basically a consequence of
> the former. There are two warnings that will show up in the log once the queue
> is more than 50% full and another one if it is more than 75% full. The biggest
> threat to run into a full queue are probably backends that are LISTENing and
> are idle in transaction.
>
>
> I have added a function pg_listening() which just contains the names of the
> channels that a backend is listening to.
>
>
> I especially invite people who know more about the transactional stuff than I
> do to take a close look at what I have done regarding notification visibility.
>
>
> One open question regarding the payload is if we need to limit it to ASCII to
> not risk conversion issues between different backend character sets?
>
>
> The second open issue is what we should do regarding 2PC. These options have
> been brought up so far:
>
>  1) allow NOTIFY in 2PC but it can happen that the transaction needs to be
>     rolled back if the queue is full
>  2) disallow NOTIFY in 2PC alltogether
>  3) put notifications to the queue on PREPARE TRANSACTION and make backends not
>     advance their pointers further than those notifications but wait for the
>     2PC transaction to commit. 2PC transactions would never fail but you
>     effectively stop the notification system until the 2PC transaction commits.
>
> Comments?

Joachim - This no longer applies - please rebase, repost, and add a
link to the new version to the commitfest app.

Jeff - Do you want to continue reviewing this for the next CommitFest,
or hand off to another reviewer?

Thanks,

...Robert