pgsql: Move buffer I/O and content LWLocks out of the main tranche.

Move buffer I/O and content LWLocks out of the main tranche.

Move the content lock directly into the BufferDesc, so that locking and
pinning a buffer touches only one cache line rather than two. Adjust
the definition of BufferDesc slightly so that this doesn't make the
BufferDesc any larger than one cache line (at least on platforms where
a spinlock is only 1 or 2 bytes).

We can't fit the I/O locks into the BufferDesc and stay within one
cache line, so move those to a completely separate tranche. This
leaves a relatively limited number of LWLocks in the main tranche, so
increase the padding of those remaining locks to a full cache line,
rather than allowing adjacent locks to share a cache line, hopefully
reducing false sharing.

Performance testing shows that these changes make little difference
on laptop-class machines, but help significantly on larger servers,
especially those with more than 2 sockets.

Andres Freund, originally based on an earlier patch by Simon Riggs.
Review and cosmetic adjustments (including heavy rewriting of the
comments) by me.

Branch
------
master

Details
-------
http://git.postgresql.org/pg/commitdiff/6150a1b08a9fe7ead2b25240be46dddeae9d98e1

Modified Files
--------------
src/backend/storage/buffer/buf_init.c | 61 +++++++++++++++++++++++++++------
src/backend/storage/buffer/bufmgr.c | 57 +++++++++++++++---------------
src/backend/storage/lmgr/lwlock.c | 15 ++++----
src/include/storage/buf_internals.h | 23 +++++++++----
src/include/storage/lwlock.h | 61 ++++++++++++++++++++++++---------
src/tools/pgindent/typedefs.list | 1 +
6 files changed, 151 insertions(+), 67 deletions(-)