Re: High SYS CPU - need advise

On Thu, Nov 15, 2012 at 2:20 PM, Vlad <marchenko(at)gmail(dot)com> wrote:
> Merlin,
>
> this is not my report, probably from a thread that I've referenced as having
> a common symptoms. Here is info about my db:
>
>
> Postgresql 9.1.6.
> Postgres usually has 400-500 connected clients, most of them are idle.
> Database is over 1000 tables (across 5 namespaces), taking ~150Gb on disk.
> We have mostly select statements (joins across few tables), using indexes
> and resulting in a small number of records returned.
>
> So mostly small lookups across few tables joining by a primary key / index.
> Also, I just perform an experiment - I switched our app over to using hot
> PITR instead of master and it experienced the same problem. So since PITR db
> can only perform read-only queries, there is no write-locks (except maybe
> when pitr is playing wal records from the master?), nevertheless SYS CPU
> jumped sky.

yeah. ok, nest steps:
*) can you confirm that postgres process is using high cpu (according
to top) during stall time
*) if, so, please strace that process and save some of the log
*) you're using a 'bleeding edge' kernel. so we must be suspicious of
a regression there, particularly in the scheduler.
*) I am suspicious of spinlock issue. so, if we can't isolate the
problem, is running a hand complied postgres a possibility (for lock
stats)?
*) what is the output of this:
echo /proc/sys/vm/zone_reclaim_mode

merlin