Re: Linux: more cores = less concurrency.

--- On Tue, 12/4/11, Greg Smith <greg(at)2ndquadrant(dot)com> wrote:

> From: Greg Smith <greg(at)2ndquadrant(dot)com>
> Subject: Re: [PERFORM] Linux: more cores = less concurrency.
> To: "Kevin Grittner" <Kevin(dot)Grittner(at)wicourts(dot)gov>
> Cc: david(at)lang(dot)hm, "Steve Clark" <sclark(at)netwolves(dot)com>, "Glyn Astill" <glynastill(at)yahoo(dot)co(dot)uk>, "Joshua D. Drake" <jd(at)commandprompt(dot)com>, "Scott Marlowe" <scott(dot)marlowe(at)gmail(dot)com>, pgsql-performance(at)postgresql(dot)org
> Date: Tuesday, 12 April, 2011, 18:00
> Kevin Grittner wrote:
> > Glyn Astill <glynastill(at)yahoo(dot)co(dot)uk>
> wrote:
> >   
> >> Results from Greg Smiths stream_scaling test are
> here:
> >>
> >> http://www.privatepaste.com/4338aa1196
> >>     
> >  Well, that pretty much clinches it.  Your
> RAM access tops out at 16
> > processors.  It appears that your processors are
> spending most of
> > their time waiting for and contending for the RAM
> bus.
> >   
>
> I've pulled Glyn's results into https://github.com/gregs1104/stream-scaling so they're
> easy to compare against similar processors, his system is
> the one labled 4 X X7550.  I'm hearing this same story
> from multiple people lately:  these 32+ core servers
> bottleneck on aggregate memory speed with running PostgreSQL
> long before the CPUs are fully utilized.  This server
> is close to maximum memory utilization at 8 cores, and the
> small increase in gross throughput above that doesn't seem
> to be making up for the loss in L1 and L2 thrashing from
> trying to run more.  These systems with many cores can
> only be used fully if you have a program that can work
> efficiency some of the time with just local CPU
> resources.  That's very rarely the case for a database
> that's moving 8K pages, tuple caches, and other forms of
> working memory around all the time.
>
>
> > I have gotten machines in where moving a jumper,
> flipping a DIP
> > switch, or changing BIOS options from the default made
> a big
> > difference.  I'd be looking at the manuals for my
> motherboard and
> > BIOS right now to see what options there might be to
> improve that
>
> I already forwarded Glyn a good article about tuning these
> Dell BIOSs in particular from an interesting blog series
> others here might like too:
>
> http://bleything.net/articles/postgresql-benchmarking-memory.html
>
> Ben Bleything is doing a very thorough walk-through of
> server hardware validation, and as is often the case he's
> already found one major problem with the vendor config he
> had to fix to get expected results.
>

Thanks Greg. I've been through that post, but unfortunately there's no settings that make a difference.

However upon further investigation and looking at the manual for the R910 here

http://support.dell.com/support/edocs/systems/per910/en/HOM/HTML/install.htm#wp1266264

I've discovered we only have 4 of the 8 memory risers, and the manual states that in this configuration we are running in "Power Optimized" mode, rather than "Performance Optimized".

We've got two of these machines, so I've just pulled all the risers from one system, removed half the memory as indicated by that document from Dell above, and now I'm seeing almost double the throughput.