Re: Tuning guidelines for server with 256GB of RAM and SSDs?

On Thu, Jul 7, 2016 at 10:27 AM, Merlin Moncure <mmoncure(at)gmail(dot)com> wrote:
> On Wed, Jul 6, 2016 at 4:48 PM, Scott Marlowe <scott(dot)marlowe(at)gmail(dot)com> wrote:
>> On Wed, Jul 6, 2016 at 12:13 PM, Merlin Moncure <mmoncure(at)gmail(dot)com> wrote:
>>> Disabling write back cache for write heavy database loads will will
>>> destroy it in short order due to write amplication and will generally
>>> cause it to underperform hard drives in my experience.
>>
>> Interesting. We found our best performance with a RAID-5 of 10 800GB
>> SSDs (Intel 3500/3700 series) that we got MUCH faster performance with
>> all write caching turned off on our LSI MEgaRAID controllers. We went
>> from 3 to 4ktps to 15 to 18ktps. And after a year of hard use we still
>> show ~90% life left (these machines handle thousands of writes per
>> second in real use) It could be that the caching was getting in the
>> way of RAID calcs or some other issue. With RAID-1 I have no clue what
>> the performance will be with write cache on or off.
>
> Right -- by that I meant disabling the write back cache on the drive
> itself, so that all writes are immediately flushed. Disabling write
> back on the raid controller should be the right choice; each of these
> drives essentially is a 'caching raid controller' for all intents and
> purposes. Hardware raid controllers are engineered around performance
> and reliability assumptions that are no longer correct in an SSD
> world. Personally I would have plugged the drives directly to the
> motherboard (assuming it's a got enough lanes) and mounted the raid
> against mdadm and compared.

Oh yeah definitely. And yea we've found that mdadm and raw HBAs work
better than most RAID controllers for SSDs.