Re: 60 core performance with 9.3

I've been assisting Mark with the benchmarking of these new servers.

The drop off in both throughput and CPU utilisation that we've been
observing as the client count increases has let me to investigate which
lwlocks are dominant at different client counts.

I've recompiled postgres with Andres LWLock improvements, Kevin's
libnuma patch and with LWLOCK_STATS enabled.

The LWLOCK_STATS below suggest that ProcArrayLock might be the main
source of locking that's causing throughput to take a dive as the client
count increases beyond the core count.

wal_buffers = 256MB
checkpoint_segments = 1920
wal_sync_method = open_datasync

pgbench -s 2000 -T 600

Results:

clients | tps
---------+---------
6 | 9490
12 | 17558
24 | 25681
48 | 41175
96 | 48954
192 | 31887
384 | 15564

LWLOCK_STATS at 48 clients

Lock | Blk | SpinDelay | Blk % | SpinDelay %
--------------------+----------+-----------+-------+-------------
BufFreelistLock | 31144 | 11 | 1.64 | 1.62
ShmemIndexLock | 192 | 1 | 0.01 | 0.15
OidGenLock | 32648 | 14 | 1.72 | 2.06
XidGenLock | 35731 | 18 | 1.88 | 2.64
ProcArrayLock | 291121 | 215 | 15.36 | 31.57
SInvalReadLock | 32136 | 13 | 1.70 | 1.91
SInvalWriteLock | 32141 | 12 | 1.70 | 1.76
WALBufMappingLock | 31662 | 15 | 1.67 | 2.20
WALWriteLock | 825380 | 45 | 36.31 | 6.61
CLogControlLock | 583458 | 337 | 26.93 | 49.49



LWLOCK_STATS at 96 clients

Lock | Blk | SpinDelay | Blk % | SpinDelay %
--------------------+----------+-----------+-------+-------------
BufFreelistLock | 62954 | 12 | 1.54 | 0.27
ShmemIndexLock | 62635 | 4 | 1.54 | 0.09
OidGenLock | 92232 | 22 | 2.26 | 0.50
XidGenLock | 98326 | 18 | 2.41 | 0.41
ProcArrayLock | 928871 | 3188 | 22.78 | 72.57
SInvalReadLock | 58392 | 13 | 1.43 | 0.30
SInvalWriteLock | 57429 | 14 | 1.41 | 0.32
WALBufMappingLock | 138375 | 14 | 3.39 | 0.32
WALWriteLock | 1480707 | 42 | 36.31 | 0.96
CLogControlLock | 1098239 | 1066 | 26.93 | 27.27



LWLOCK_STATS at 384 clients

Lock | Blk | SpinDelay | Blk % | SpinDelay %
--------------------+----------+-----------+-------+-------------
BufFreelistLock | 184298 | 158 | 1.93 | 0.03
ShmemIndexLock | 183573 | 164 | 1.92 | 0.03
OidGenLock | 184558 | 173 | 1.93 | 0.03
XidGenLock | 200239 | 213 | 2.09 | 0.04
ProcArrayLock | 4035527 | 579666 | 42.22 | 98.62
SInvalReadLock | 182204 | 152 | 1.91 | 0.03
SInvalWriteLock | 182898 | 137 | 1.91 | 0.02
WALBufMappingLock | 219936 | 215 | 2.30 | 0.04
WALWriteLock | 3172725 | 457 | 24.67 | 0.08
CLogControlLock | 1012458 | 6423 | 10.59 | 1.09

The same test done with a readonly workload show virtually no SpinDelay
at all.

Any thoughts or comments on these results are welcome!

Regards,
Matt.