Re: Sudden crazy high CPU usage

Thanks, this seems to persist after a reboot of the server though, and I have never in my server’s 3 months life time experienced anything like it.

Niels Kristian Schjødt
Co-founder & Developer

E-Mail: nielskristian(at)autouncle(dot)com
Mobile: 0045 28 73 04 93

www.autouncle.com
Follow us: Facebook | Google+ | LinkedIn | Twitter
Get app for: iPhone & iPad | Android

Den 31/03/2014 kl. 15.47 skrev Merlin Moncure <mmoncure(at)gmail(dot)com>:

> On Mon, Mar 31, 2014 at 5:25 AM, Niels Kristian Schjødt
> <nielskristian(at)autouncle(dot)com> wrote:
>> I'm running postgresql 9.3 on a production server. An hour ago, out of the "blue", I ran into an issue I have never encountered before: my server started to use CPU as crazy. The server is a standard ubuntu 12.04 LTE installation running only Postgres and Redis.
>>
>> The incident can be seen on the in numbers below:
>>
>> https://s3-eu-west-1.amazonaws.com/autouncle-public/other/cpu.png
>>
>> I imidiatly took a look at pg_stat_activity but nothing in there seemed suspicious. I also had a look at the postgres log, but nothing was in there too. I have pg_stat_statements running, so I reseted that one, and nothing really suspicious occurred in there, expect for the fact, that all queries were taking 100x times longer than usual.
>>
>> I have tried the following with no luck:
>>
>> * Restart clients connecting to the db
>> * Restart postgres
>> * Restart the whole server
>>
>> I have run memory tests on the server as well, and nothing seems to be wrong.
>>
>> No changes in any software running on the servers has been made within the last 24 hours.
>>
>> The question is: I have a streaming replication server running, which I have now done a failover to, and it runs fine. However I still have no clue why my master suddenly has become so CPU consuming, and how I can debug / trace it further down?
>
> Using linux 6? One possible culprit is "Transparent Huge Page
> Compaction". It tends to hit severs with a lot of memory, especially
> if they've configured a lot of shared buffers. Google it a for a lot
> of info.
>
> There may be other issues masquerading as this one but it's the first
> thing to rule out. Symptoms are very high cpu utilization and poor
> performance that strikes without warning and then resolves also
> without warning (typically seconds or minutes after the event).
>
> For starters, take a look at the value of:
>
> /sys/kernel/mm/redhat_transparent_hugepage/enabled
>
> And do some due diligence research.
>
> merlin