Further info on LWLock behavior

I added some code to the backend to count the number of LWLockAcquire
calls and the number of resultant process blocks (IpcSemaphoreLocks)
across all PG processes. Here are the results for a pgbench run with
scale factor 500, number of clients 10, proposed LWLock patch installed
on that 4-way Linux box.

LWLock acquires blocks fraction

BufMgrLock 1238892 60273 0.0486507298457008
LockMgrLock 690150 83250 0.120625950880243
OidGenLock 10004 3 0.000299880047980808
XidGenLock 60019 29 0.000483180326230027
ShmemIndexLock 180 0 0
SInvalLock 817017 1940 0.00237449159564611
FreeSpaceLock 290 0 0
WALInsertLock 80039 6139 0.0767001086970102
WALWriteLock 10194 1180 0.115754365312929
ControlFileLock 141 0 0
CLogControlLock 25366 56 0.0022076795710794
buf cxt locks 930006 172 0.000184945043365312
buf io locks 31859 142 0.00445713926990803
clog buf locks 1880 2 0.00106382978723404

Interesting data, eh? In particular, it seems my previous opinion
that BufMgrLock was the main issue is all wet: the LockMgrLock accounts
for more blockages despite being locked fewer times. AFAICS this must
mean that the average time of holding LockMgrLock is larger than the
average time of holding BufMgrLock, and that we ought to look at how
to reduce that. The WAL locks also seem to have disproportionately
large blocking percentages.

regards, tom lane