Re: Listen / Notify - what to do when the queue is full

On Wed, Feb 3, 2010 at 2:05 AM, Jeff Davis <pgsql(at)j-davis(dot)com> wrote:
> The original comment was a part of the NotifyStmt case, and I don't
> think we can support NOTIFY issued on a standby system -- surely there's
> no way for the standby to communicate the notification to the master.
> Anyway, this is getting a little sidetracked; I don't think we need to
> worry about HS right now.

True but I was not talking about moving any notifications to different
servers. Clients listening on one server should receive the
notifications from NOTIFYs executed on this server, no matter if it is
a standby or the master server. It seems to me that currently we
cannot support Listen/Notify on the Hot Standby because it calls
GetCurrentTransactionId() which is not available there. On the other
hand this is just done because we fear that a transaction could
rollback while actually committing. On the read-only standby however a
transaction that commits always commits sucessfully because it has not
changed any data (right?)...

>> The second doubt I had is about the truncation behavior of slru. ISTM
>> that it doesn't truncate at the end of the page range once the head
>> pointer has already wrapped around.
>> There is the following comment in slru.c describing this fact:
>>
>>     /*
>>      * While we are holding the lock, make an important safety check: the
>>      * planned cutoff point must be <= the current endpoint page. Otherwise we
>>      * have already wrapped around, and proceeding with the truncation would
>>      * risk removing the current segment.
>>      */
>>
>> I wanted to check if we can do anything about it and if we need to do
>> anything at all...
>
> I'll have to take a look in more detail.

This is still kind of an open item but it's an slru issue and should
also be true for other functionality that uses slru queues.

>> Now the question is, should we forbid NOTIFY for 2PC
>> altogether only because in the unlikely event of a full queue we
>> cannot guarantee that we can commit the transaction?
>
> I believe that was the consensus in the thread. The people who use 2PC
> are the people that want the most rock-solid guarantees. I don't like
> forbidding feature combinations, but I don't see a good alternative.

If this was consensus, then fine... I have forbidden it in the latest patch now.

>> One solution is to treat a 2PC transaction like a backend with its own
>> pointer to the queue. As long as the prepared transaction is not
>> committed, its pointer does not move and so we don't move forward the
>> global tail pointer. Here the NOTIFYs sent by the 2PC transaction are
>> already in the queue and the transaction can always commit. The
>> drawback of this idea is that if you forget to commit the prepared
>> transaction and leave it around uncommitted, your queue will fill up
>> inevitably because you do not truncate anymore...
>
> There's also a problem if the power goes out, you restart, and then the
> queue fills up before you COMMIT PREPARED.

This I don't understand... If power goes out and we restart, we'd
first put all notifications from the prepared transactions into the
queue. We know that they fit because they have fit earlier as well (we
wouldn't allow user connections until we have worked through all 2PC
state files).

> I'm worried that this creates the opposite problem: that a LISTEN
> transaction might commit before a NOTIFY transaction, and yet miss the
> notification.

(see my other email on this item...)

>> I have also included Arnaud Betremieux's send_notify function. Should
>> we really call the function send_notify? What about
>> pg_send_notification or just pg_notify?
>
> Agreed: pg_notify sounds good to me.

Function name changed.

There was another problem that the slru files did not all get deleted
at server restart, which is fixed now.

Regarding the famous ASCII-restriction open item I have now realized
what I haven't thought of previously: notifications are not
transferred between databases, they always stay in one database. Since
libpq does the conversion between server and client encoding, it is
questionable if we really need to restrict this at all... But I am not
an encoding expert so whoever feels like he can confirm or refute
this, please speak up.

Joachim