Re: [RFC] CREATE QUEUE (log-only table) for londiste/pgQ ccompatibility

On Wed, Oct 17, 2012 at 4:25 PM, Josh Berkus <josh(at)agliodbs(dot)com> wrote:
>> It is not meant to be a full implementation of application level queuing
>> system though but just the capture, persisting and distribution parts
>>
>> Using this as an "application level queue" needs a set of interface
>> functions to extract the events and also to keep track of the processed
>> events. As there is no general consensus what these shoul be (like if
>> processing same event twice is allowed) this part is left for specific
>> queue consumer implementations.
>
> Well, but AFAICT, you've already prohibited features through your design
> which are essential to application-level queues, and are implemented by,
> for example, pgQ.
>
> 1. your design only allows the queue to be read on replicas, not on the
> node where the item was inserted.
>
> 2. if you can't UPDATE or DELETE queue items -- or LOCK them -- how on
> earth would a client know which items they have executed and which they
> haven't?
>
> 3. Double-down on #2 in a multithreaded environment.
>
> For an application-level queue, the base functionality is:
>
> ADD ITEM
> READ NEXT (#) ITEM(S)
> LOCK ITEM
> DELETE ITEM
>
> More sophisticated an useful queues also allow:
>
> READ NEXT UNLOCKED ITEM
> LOCK NEXT UNLOCKED ITEM
> UPDATE ITEM
> READ NEXT (#) UNSEEN ITEM(S)
>
> The design you describe seems to prohibit pretty much all of the above
> operations after READ NEXT. This makes it completely useless as a
> application-level queue.
>
> And, for that matter, if your new queue only accepts INSERTs, why not
> just improve LISTEN/NOTIFY so that it's readable on replicas? What does
> this design buy you that that doesn't?

I've read the whole thread, but I still don't see that anyone's said
it better than this, and I agree with these comments. (I don't find
them ad hominem, either.)

It's also worth noting that in order to be useful, this feature
intrinsically requires the logical replication stuff to be committed
first. It's not entirely clear that there's not enough time to get
logical replication committed for 9.3, and the chances of getting any
follow-on features getting committed as well seems remote. Besides
the shortness of the time, I think all experience has shown that it's
best not to rush into the design of follow-on features before we've
got the basic feature well nailed down. This certainly can't be said
of logical replication at this point. Andres seems to be making good
progress and I'm grateful for his work on it, but I think there's a
lot left to do before that one is in the bag (as I think Andres would
agree).

--
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company