[PATCH] Improve performance of NOTIFY over many databases (v2)

Hoi hackers,

Here is a reworked version of the previous patches.

The original three patches have been collapsed into one as given the
changes discussed it didn't make sense to keep them separate. There
are now two patches (the third is just to help with testing):

Patch 1: Tracks the listening backends in a list so non-listening
backends can be quickly skipped over. This is separate because it's
orthogonal to the rest of the changes and there are other ways to do
this.

Patch 2: This is the meat of the change. It implements all the
suggestions discussed:

- The queue tail is now only updated lazily, whenever the notify queue
moves to a new page. This did require a new global to track this state
through the transaction commit, but it seems worth it.

- Only backends for the current database are signalled when a
notification is made

- Slow backends are woken up one at a time rather than all at once

- A backend is allowed to lag up to 4 SLRU pages behind before being
signalled. This is a tradeoff between how often to get woken up verses
how much work to do once woken up.

- All the relevant comments have been updated to describe the new
algorithm. Locking should also be correct now.

This means in the normal case where listening backends get a
notification occasionally, no-one will ever be considered slow. An
exclusive lock for cleanup will happen about once per SLRU page.
There's still the exclusive locks on adding notifications but that's
unavoidable.

One minor issue is that pg_notification_queue_usage() will now return
a small but non-zero number (about 3e-6) even when nothing is really
going on. This could be fixed by having it take an exclusive lock
instead and updating to the latest values but that barely seems worth
it.

Performance-wise it's even better than my original patches, with about
20-25% reduction in CPU usage in my test setup (using the test script
sent previously).

Here is the log output from my postgres, where you see the signalling in action:

------
16:42:48.673 [10188] martijn(at)test_131 DEBUG: PreCommit_Notify
16:42:48.673 [10188] martijn(at)test_131 DEBUG: NOTIFY QUEUE = (74,896)...(79,0)
16:42:48.673 [10188] martijn(at)test_131 DEBUG: backendTryAdvanceTail -> true
16:42:48.673 [10188] martijn(at)test_131 DEBUG: AtCommit_Notify
16:42:48.673 [10188] martijn(at)test_131 DEBUG: ProcessCompletedNotifies
16:42:48.673 [10188] martijn(at)test_131 DEBUG: backendTryAdvanceTail -> false
16:42:48.673 [10188] martijn(at)test_131 DEBUG: asyncQueueAdvanceTail
16:42:48.673 [10188] martijn(at)test_131 DEBUG: waking backend 137 (pid 10055)
16:42:48.673 [10055] martijn(at)test_067 DEBUG: ProcessIncomingNotify
16:42:48.673 [10187] martijn(at)test_131 DEBUG: ProcessIncomingNotify
16:42:48.673 [10055] martijn(at)test_067 DEBUG: asyncQueueAdvanceTail
16:42:48.673 [10055] martijn(at)test_067 DEBUG: waking backend 138 (pid 10056)
16:42:48.673 [10187] martijn(at)test_131 DEBUG: ProcessIncomingNotify: done
16:42:48.673 [10055] martijn(at)test_067 DEBUG: ProcessIncomingNotify: done
16:42:48.673 [10056] martijn(at)test_067 DEBUG: ProcessIncomingNotify
16:42:48.673 [10056] martijn(at)test_067 DEBUG: asyncQueueAdvanceTail
16:42:48.673 [10056] martijn(at)test_067 DEBUG: ProcessIncomingNotify: done
16:42:48.683 [9991] martijn(at)test_042 DEBUG: Async_Notify(changes)
16:42:48.683 [9991] martijn(at)test_042 DEBUG: PreCommit_Notify
16:42:48.683 [9991] martijn(at)test_042 DEBUG: NOTIFY QUEUE = (75,7744)...(79,32)
16:42:48.683 [9991] martijn(at)test_042 DEBUG: AtCommit_Notify
-----

Have a nice weekend.
--
Martijn van Oosterhout <kleptog(at)gmail(dot)com> http://svana.org/kleptog/