Re: upgrade from 9.2.x to 9.3 causes significant performance degradation

On Wed, Sep 18, 2013 at 2:02 AM, Kevin Grittner <kgrittn(at)ymail(dot)com> wrote:
> Lonni J Friedman <netllama(at)gmail(dot)com> wrote:
>
>> top shows over 90% of the load is in sys space. vmstat output
>> seems to suggest that its CPU bound (or bouncing back & forth):
>
> Can you run `perf top` during an episode and see what kernel
> functions are using all that CPU?

I take back what I said earlier. While the master is currently back
to normal performance, the two hot standby slaves are still churning
something awful.

If I run 'perf top' on either slave, after a few seconds, these are
consistently the top three in the list:
84.57% [kernel] [k] _spin_lock_irqsave
6.21% [unknown] [.] 0x0000000000659f60
4.69% [kernel] [k] compaction_alloc

>
> This looks similar to cases I've seen of THP defrag going wild.
> Did the OS version or configuration change? Did the PostgreSQL
> memory settings (like shared_buffers) change?

I think you're onto something here with respect to THP defrag going
wild. I set /sys/kernel/mm/transparent_hugepage/defrag to 'never' and
immediately the load dropped on both slaves from over 5.00 to under
1.00.

So this raises the question, is this a kernel bug, or is there some
other solution to the problem?
Also, seems weird that the problem didn't happen until I switched from
9.2 to 9.3. Is it possible this is somehow related to the change from
using SysV shared memory to using Posix shared memory and mmap for
memory management?