Re: CPU load spikes when CentOS tries to reclaim 'cached' memory

From: Deron <fecastle(at)gmail(dot)com>
To: Merlin Moncure <mmoncure(at)gmail(dot)com>
Cc: Jeff Janes <jeff(dot)janes(at)gmail(dot)com>, vlasmarias <vlasmarias(at)contigo(dot)com>, "pgsql-performance(at)postgresql(dot)org" <pgsql-performance(at)postgresql(dot)org>
Subject: Re: CPU load spikes when CentOS tries to reclaim 'cached' memory
Date: 2014-06-05 19:47:34
Message-ID: CAF3Lvs5HPj9=8UV8bFuXqXk5G-=8spg9MoYkkt11oZd6SXFe4g@mail.gmail.com
Views: Raw Message | Whole Thread | Download mbox | Resend email
Thread:
Lists: pgsql-performance

We saw very similar issues with a CentOS server with 40 cores (32
virtualized) when moving from a physical server to a virtual server (I
think it had 128GB RAM). Never had the problem on a physical server. We
checked the same things as noted here, but never found a bug. We really
thought it had something to do with NUMA zone reclaim, but could never
prove that. In our case it was all kernel time in the guest, all CPUs at
100%. Sometimes it would last for a few seconds or minutes. Sometimes we
would go days without a problem, and then it would completely tank.

If you figure out what is going on, I would like to know (especially if it
is virtualized).

Deron

On Thu, Jun 5, 2014 at 12:23 PM, Merlin Moncure <mmoncure(at)gmail(dot)com> wrote:

> On Thu, Jun 5, 2014 at 10:58 AM, Jeff Janes <jeff(dot)janes(at)gmail(dot)com> wrote:
> > This sounds like a kernel problem, probably either the zone reclaim
> issue,
> > or the transparent huge pages issue.
>
> I at first thought maybe same, but I don't think THP was introduced
> until 2.6.38...OP is running 2.6.32-431.11.2.el6.x86_6. Maybe it's
> NUMA related, but would not be idiomatic of NUMA issues as I
> understand them (poor memory utilization/high IO utilization). Would
> be a very cheap/easy thing to try though.
>
> Is this server virtualized?
>
> merlin
>
>
> --
> Sent via pgsql-performance mailing list (pgsql-performance(at)postgresql(dot)org)
> To make changes to your subscription:
> http://www.postgresql.org/mailpref/pgsql-performance
>

In response to

Responses

Browse pgsql-performance by date

  From Date Subject
Next Message Merlin Moncure 2014-06-05 19:57:13 Re: CPU load spikes when CentOS tries to reclaim 'cached' memory
Previous Message Weinzierl Stefan 2014-06-05 19:36:08 Seqscan on big table, when an Index-Usage should be possible