LISTEN, select/poll question/clarification

From: Israel Brewster <israel(at)ravnalaska(dot)net>
To: psycopg(at)postgresql(dot)org
Subject: LISTEN, select/poll question/clarification
Date: 2017-12-05 18:31:32
Message-ID: 433B2364-51CA-47F8-93B6-AC5CF5849DFC@ravnalaska.net
Views: Raw Message | Whole Thread | Download mbox | Resend email
Thread:
Lists: psycopg

I am using the PostgreSQL PubSub feature with psycopg2 and gevent in the following manner (dbconn is, of course, a psycopg2 connection object on which LISTEN has been called):

while True:
if gevent.select.select([dbconn], [], []) != ([], [], []):
dbconn.poll()
gevent.spawn(process_result,dbconn.notifies)

Which works fine. Now my understanding of how this all works is that when dbconn.poll() is called, it should pull in *all* pending NOTIFYs and append them to the dbconn.notifies object. So lets say I have NOTIFYs coming in at a rate of 1 per second. I would think that if I added a "gevent.sleep(5)" to the end of the above while loop, then each time through I should have 5 notifies in the dbconn.notifies list, since it has been 5 seconds since I last checked. However, that doesn't appear to be the case - rather, no mater how long a sleep I put in, I still only get *one* item in the notifies list, making me think that I am missing data.

Can someone explain why this is? Why am I not getting 5 at a time under that scenario?

In case someone was wondering, the reasoning behind adding the sleep() lies in that process_result function. Due to the looping it contains, it can process, say, 5 data points in one call much more efficiently than it can process those same 5 data points in 5 calls of 1 data point each. So rather than run it every time a data point comes in, I would like to let the data "pile up" as it were for a short period of time before processing. I was thinking I could easily accomplish this by simply waiting some appropriate period of time between poll() ing the database for new NOTIFYs, but apparently that's not working. The select() is there because the data isn't coming in at a regular speed like in my example, but rather it could come in much faster, or not at all for a while, depending on the time of day.
-----------------------------------------------
Israel Brewster
Systems Analyst II
Ravn Alaska
5245 Airport Industrial Rd
Fairbanks, AK 99709
(907) 450-7293
-----------------------------------------------

Responses

Browse psycopg by date

  From Date Subject
Next Message Daniele Varrazzo 2017-12-05 19:25:57 Re: LISTEN, select/poll question/clarification
Previous Message Murtuza Zabuawala 2017-11-22 06:05:57 Re: pgpass option is not working with hostaddr parameter