Re: New server: SSD/RAID recommendations?

----- Original Message -----
> From: "Graeme B. Bell" <graeme(dot)bell(at)nibio(dot)no>
> To: "Mkrtchyan, Tigran" <tigran(dot)mkrtchyan(at)desy(dot)de>
> Cc: "Graeme B. Bell" <graeme(dot)bell(at)nibio(dot)no>, "Steve Crawford" <scrawford(at)pinpointresearch(dot)com>, "Wes Vaske (wvaske)"
> <wvaske(at)micron(dot)com>, "pgsql-performance" <pgsql-performance(at)postgresql(dot)org>
> Sent: Tuesday, July 7, 2015 12:38:10 PM
> Subject: Re: [PERFORM] New server: SSD/RAID recommendations?

> I am unsure about the performance side but, ZFS is generally very attractive to
> me.
>
> Key advantages:
>
> 1) Checksumming and automatic fixing-of-broken-things on every file (not just
> postgres pages, but your scripts, O/S, program files).
> 2) Built-in lightweight compression (doesn't help with TOAST tables, in fact
> may slow them down, but helpful for other things). This may actually be a net
> negative for pg so maybe turn it off.
> 3) ZRAID mirroring or ZRAID5/6. If you have trouble persuading someone that it's
> safe to replace a RAID array with a single drive... you can use a couple of
> NVMe SSDs with ZFS mirror or zraid, and get the same availability you'd get
> from a RAID controller. Slightly better, arguably, since they claim to have
> fixed the raid write-hole problem.
> 4) filesystem snapshotting
>
> Despite the costs of checksumming etc., I suspect ZRAID running on a fast CPU
> with multiple NVMe drives will outperform quite a lot of the alternatives, with
> great data integrity guarantees.

We are planing to have a test setup as well. For now I have single NVMe SSD on my
test system:

# lspci | grep NVM
85:00.0 Non-Volatile memory controller: Samsung Electronics Co Ltd NVMe SSD Controller 171X (rev 03)

# mount | grep nvm
/dev/nvme0n1p1 on /var/lib/pgsql/9.5 type ext4 (rw,noatime,nodiratime,data=ordered)

and quite happy with it. We have write heavy workload on it to see when it will
break. Postgres Performs very well. About x2.5 faster than with regular disks
with a single client and almost linear with multiple clients (picture attached.
On Y number of high level op/s our application does, X number of clients). The
setup is used last 3 months. Looks promising but for production we need to
to have disk size twice as big as on the test system. Until today, I was
planning to use a RAID10 with a HW controller...

Related to ZFS. We use ZFSonlinux and behaviour is not as good as with solaris.
Let's re-phrase it: performance is unpredictable. We run READZ2 with 30x3TB disks.

Tigran.

>
> Haven't built one yet. Hope to, later this year. Steve, I would love to know
> more about how you're getting on with your NVMe disk in postgres!
>
> Graeme.
>
> On 07 Jul 2015, at 12:28, Mkrtchyan, Tigran <tigran(dot)mkrtchyan(at)desy(dot)de> wrote:
>
>> Thanks for the Info.
>>
>> So if RAID controllers are not an option, what one should use to build
>> big databases? LVM with xfs? BtrFs? Zfs?
>>
>> Tigran.
>>
>> ----- Original Message -----
>>> From: "Graeme B. Bell" <graeme(dot)bell(at)nibio(dot)no>
>>> To: "Steve Crawford" <scrawford(at)pinpointresearch(dot)com>
>>> Cc: "Wes Vaske (wvaske)" <wvaske(at)micron(dot)com>, "pgsql-performance"
>>> <pgsql-performance(at)postgresql(dot)org>
>>> Sent: Tuesday, July 7, 2015 12:22:00 PM
>>> Subject: Re: [PERFORM] New server: SSD/RAID recommendations?
>>
>>> Completely agree with Steve.
>>>
>>> 1. Intel NVMe looks like the best bet if you have modern enough hardware for
>>> NVMe. Otherwise e.g. S3700 mentioned elsewhere.
>>>
>>> 2. RAID controllers.
>>>
>>> We have e.g. 10-12 of these here and e.g. 25-30 SSDs, among various machines.
>>> This might give people idea about where the risk lies in the path from disk to
>>> CPU.
>>>
>>> We've had 2 RAID card failures in the last 12 months that nuked the array with
>>> days of downtime, and 2 problems with batteries suddenly becoming useless or
>>> suddenly reporting wildly varying temperatures/overheating. There may have been
>>> other RAID problems I don't know about.
>>>
>>> Our IT dept were replacing Seagate HDDs last year at a rate of 2-3 per week (I
>>> guess they have 100-200 disks?). We also have about 25-30 Hitachi/HGST HDDs.
>>>
>>> So by my estimates:
>>> 30% annual problem rate with RAID controllers
>>> 30-50% failure rate with Seagate HDDs (backblaze saw similar results)
>>> 0% failure rate with HGST HDDs.
>>> 0% failure in our SSDs. (to be fair, our one samsung SSD apparently has a bug
>>> in TRIM under linux, which I'll need to investigate to see if we have been
>>> affected by).
>>>
>>> also, RAID controllers aren't free - not just the money but also the management
>>> of them (ever tried writing a complex install script that interacts work with
>>> MegaCLI? It can be done but it's not much fun.). Just take a look at the
>>> MegaCLI manual and ask yourself... is this even worth it (if you have a good
>>> MTBF on an enterprise SSD).
>>>
>>> RAID was meant to be about ensuring availability of data. I have trouble
>>> believing that these days....
>>>
>>> Graeme Bell
>>>
>>>
>>> On 06 Jul 2015, at 18:56, Steve Crawford <scrawford(at)pinpointresearch(dot)com> wrote:
>>>
>>>>
>>>> 2. We don't typically have redundant electronic components in our servers. Sure,
>>>> we have dual power supplies and dual NICs (though generally to handle external
>>>> failures) and ECC-RAM but no hot-backup CPU or redundant RAM banks and...no
>>>> backup RAID card. Intel Enterprise SSD already have power-fail protection so I
>>>> don't need a RAID card to give me BBU. Given the MTBF of good enterprise SSD
>>>> I'm left to wonder if placing a RAID card in front merely adds a new point of
>>>> failure and scheduled-downtime-inducing hands-on maintenance (I'm looking at
>>>> you, RAID backup battery).
>>>
>>>
>>>
>>> --
>>> Sent via pgsql-performance mailing list (pgsql-performance(at)postgresql(dot)org)
>>> To make changes to your subscription:
>>> http://www.postgresql.org/mailpref/pgsql-performance
>
>
>
> --
> Sent via pgsql-performance mailing list (pgsql-performance(at)postgresql(dot)org)
> To make changes to your subscription:
> http://www.postgresql.org/mailpref/pgsql-performance