Re: Rewrite sinval messaging to reduce contention

I wrote:
> In this thread back in January
> http://archives.postgresql.org/pgsql-performance/2008-01/msg00001.php
> we saw evidence that contention for SInvalLock was a problem,
> and we (or at least I) concluded that the sinval messaging system
> needed a nontrivial rewrite to get rid of its habit of sending everyone
> to read the queue at the same time. Here is the proposed rewrite.

After further thought, I came up with two more improvements:

* It turns out not to be too hard to apply the idea of moving multiple
messages per LWLock acquisition on the writing side too, so I did that.

* There is not actually any good reason why writers have to block
readers or vice versa, except in the infrequent case that SICleanupQueue
is needed. Writers don't look at the ProcStates that the readers are
changing. The only point of overlap is that a writer will change
maxMsgNum which readers want to look at --- but we are already assuming
in several places that reading or writing an int is atomic, and if we
assume that here too, it seems to work fine. To implement this, I split
SInvalLock into SInvalReadLock and SInvalWriteLock.

This seems to be reaching a point of diminishing returns for the
particular test case I'm using --- the TPS rate only went up from 700
to 730 or so. However, enabling LWLOCK_STATS shows that the contention
rate on the sinval locks is now completely negligible --- one block
per thousand acquisitions on SInvalWriteLock, and less than one in
10000 on SInvalReadLock. The vast majority of the LWLock contention
now comes from WALInsertLock and the LockMgr locks:

Lock # acquisitions # times blocked

SInvalReadLock 6469840 380
SInvalWriteLock 240567 163
WALInsertLock 2388805 89142
LockMgr partition locks 8253142 177715

So I think this patch accomplishes the goal of making sinval not be a
cause of contention.

Patch version 2 attached.

regards, tom lane