pgsql: Avoid the use of a separate spinlock to protect a LWLock's wait

Avoid the use of a separate spinlock to protect a LWLock's wait queue.

Previously we used a spinlock, in adition to the atomically manipulated
->state field, to protect the wait queue. But it's pretty simple to
instead perform the locking using a flag in state.

Due to 6150a1b0 BufferDescs, on platforms (like PPC) with > 1 byte
spinlocks, increased their size above 64byte. As 64 bytes are the size
we pad allocated BufferDescs to, this can increase false sharing;
causing performance problems in turn. Together with the previous commit
this reduces the size to <= 64 bytes on all common platforms.

Author: Andres Freund
Discussion: CAA4eK1+ZeB8PMwwktf+3bRS0Pt4Ux6Rs6Aom0uip8c6shJWmyg(at)mail(dot)gmail(dot)com
20160327121858(dot)zrmrjegmji2ymnvr(at)alap3(dot)anarazel(dot)de

Branch
------
master

Details
-------
http://git.postgresql.org/pg/commitdiff/008608b9d51061b1f598c197477b3dc7be9c4a64

Modified Files
--------------
src/backend/storage/lmgr/lwlock.c | 189 ++++++++++++++++++++++----------------
src/include/storage/lwlock.h | 6 +-
2 files changed, 113 insertions(+), 82 deletions(-)