sched_migration_cost for high-connection counts

Hey guys,

I recently stumbled over a Linux scheduler setting that has outright
shocked me. So, after reading through this:

http://blog.tsunanet.net/2010/11/how-long-does-it-take-to-make-context.html

it became readily apparent we were hitting the same wall. I could do a
pgbench and increase the connection count by 100 every iteration, and
eventually performance just fell off a proverbial cliff and never recovered.

For our particular systems, this barrier is somewhere around 800
processes. Select-only performance on a 3600-scale pgbench database in
cache falls from about 70k TPS to about 12k TPS after crossing that
line. Worse, sar shows over 70% CPU dedicated to system overhead.

After some fiddling around, I changed sched_migration_cost from its
default of 500000 to 5000000 and performance returned to linear scaling
immediately. It's literally night and day. Setting it back to 500000
reverts to the terrible performance.

In addition, setting the migration cost to a higher value does not
negatively affect any other performance metric I've checked. This is on
an Ubuntu 12.04 system, and I'd love if someone out there could
independently verify this, because frankly, I find it difficult to believe.

If legit, high-connection systems would benefit greatly.

--
Shaun Thomas
OptionsHouse | 141 W. Jackson Blvd. | Suite 500 | Chicago IL, 60604
312-444-8534
sthomas(at)optionshouse(dot)com

______________________________________________

See http://www.peak6.com/email_disclaimer/ for terms and conditions related to this email