Re: High SYS CPU - need advise

On Thu, Nov 15, 2012 at 4:29 PM, Alvaro Herrera
<alvherre(at)2ndquadrant(dot)com> wrote:
> Merlin Moncure escribió:
>
>> ok, excellent. reviewing the log, this immediately caught my eye:
>>
>> recvfrom(8, "\27\3\1\0@", 5, 0, NULL, NULL) = 5
>> recvfrom(8, "\327\327\nl\231LD\211\346\243(at)WW\254\244\363C\326\247\341\177\255\263~\327HDv-\3466\353"...,
>> 64, 0, NULL, NULL) = 64
>> select(0, NULL, NULL, NULL, {0, 1000}) = 0 (Timeout)
>> select(0, NULL, NULL, NULL, {0, 1000}) = 0 (Timeout)
>> select(0, NULL, NULL, NULL, {0, 1000}) = 0 (Timeout)
>> select(0, NULL, NULL, NULL, {0, 2000}) = 0 (Timeout)
>> select(0, NULL, NULL, NULL, {0, 3000}) = 0 (Timeout)
>> select(0, NULL, NULL, NULL, {0, 4000}) = 0 (Timeout)
>> select(0, NULL, NULL, NULL, {0, 6000}) = 0 (Timeout)
>> select(0, NULL, NULL, NULL, {0, 7000}) = 0 (Timeout)
>> select(0, NULL, NULL, NULL, {0, 8000}) = 0 (Timeout)
>> select(0, NULL, NULL, NULL, {0, 9000}) = 0 (Timeout)
>> semop(41713721, {{2, 1, 0}}, 1) = 0
>> lseek(295, 0, SEEK_END) = 0
>> lseek(296, 0, SEEK_END) = 8192
>>
>> this is definitely pointing to spinlock issue.
>
> I met Rik van Riel (Linux kernel hacker) recently and we chatted about
> this briefly. He strongly suggested that we should consider using
> futexes on Linux instead of spinlocks; the big advantage being that
> futexes sleep instead of spinning when contention is high. That would
> reduce the system load in this scenario.

By the way, I think you (and Rik) 100% right. We're slamming select()
where futex has much smarter wait mechanism. There are reports from
users who have switched to pthreads and reporting better scalability.
(https://groups.google.com/forum/?fromgroups=#!topic/pgsql.hackers/v8V82gznxs8)

merlin