pgsql: Change locking regimen around buffer replacement.

Change locking regimen around buffer replacement.

Previously, we used an lwlock that was held from the time we began
seeking a candidate buffer until the time when we found and pinned
one, which is disastrous for concurrency. Instead, use a spinlock
which is held just long enough to pop the freelist or advance the
clock sweep hand, and then released. If we need to advance the clock
sweep further, we reacquire the spinlock once per buffer.

This represents a significant increase in atomic operations around
buffer eviction, but it still wins on many workloads. On others, it
may result in no gain, or even cause a regression, unless the number
of buffer mapping locks is also increased. However, that seems like
material for a separate commit. We may also need to consider other
methods of mitigating contention on this spinlock, such as splitting
it into multiple locks or jumping the clock sweep hand more than one
buffer at a time, but those, too, seem like separate improvements.

Patch by me, inspired by a much larger patch from Amit Kapila.
Reviewed by Andres Freund.

Branch
------
master

Details
-------
http://git.postgresql.org/pg/commitdiff/5d7962c6797c0baae9ffb3b5b9ac0aec7b598bc3

Modified Files
--------------
src/backend/storage/buffer/README | 39 ++++++++++----------
src/backend/storage/buffer/bufmgr.c | 12 ++-----
src/backend/storage/buffer/freelist.c | 63 +++++++++++++++++++--------------
src/include/storage/buf_internals.h | 5 ++-
src/include/storage/lwlock.h | 2 +-
5 files changed, 61 insertions(+), 60 deletions(-)