Sudden crazy high CPU usage

From: Niels Kristian Schjødt <nielskristian(at)autouncle(dot)com>
To: "pgsql-performance(at)postgresql(dot)org list" <pgsql-performance(at)postgresql(dot)org>
Subject: Sudden crazy high CPU usage
Date: 2014-03-31 10:25:52
Message-ID: 24BBD031-7384-4FFF-848B-90F5664C0063@autouncle.com
Views: Raw Message | Whole Thread | Download mbox | Resend email
Thread:
Lists: pgsql-performance

I'm running postgresql 9.3 on a production server. An hour ago, out of the "blue", I ran into an issue I have never encountered before: my server started to use CPU as crazy. The server is a standard ubuntu 12.04 LTE installation running only Postgres and Redis.

The incident can be seen on the in numbers below:

https://s3-eu-west-1.amazonaws.com/autouncle-public/other/cpu.png

I imidiatly took a look at pg_stat_activity but nothing in there seemed suspicious. I also had a look at the postgres log, but nothing was in there too. I have pg_stat_statements running, so I reseted that one, and nothing really suspicious occurred in there, expect for the fact, that all queries were taking 100x times longer than usual.

I have tried the following with no luck:

• Restart clients connecting to the db
• Restart postgres
• Restart the whole server

I have run memory tests on the server as well, and nothing seems to be wrong.

No changes in any software running on the servers has been made within the last 24 hours.

The question is: I have a streaming replication server running, which I have now done a failover to, and it runs fine. However I still have no clue why my master suddenly has become so CPU consuming, and how I can debug / trace it further down?

Responses

Browse pgsql-performance by date

  From Date Subject
Next Message Shaun Thomas 2014-03-31 13:17:55 Re: Slow Count-Distinct Query
Previous Message Cora Ma 2014-03-31 08:31:39 performance degradation after launching postgres cluster using pgpool-II