Re: Linux I/O schedulers - CFQ & random seeks

On Sat, Mar 5, 2011 at 6:09 AM, Rosser Schwarz <rosser(dot)schwarz(at)gmail(dot)com> wrote:
> On Fri, Mar 4, 2011 at 10:34 AM, Glyn Astill <glynastill(at)yahoo(dot)co(dot)uk> wrote:
>> I'm wondering (and this may be a can of worms) what peoples opinions are on these schedulers?  I'm going to have to do some real world testing myself with postgresql too, but initially was thinking of switching from our current CFQ back to deadline.
>
> It was a few years ago now, but I went through a similar round of
> testing, and thought CFQ was fine, until I deployed the box.  It fell
> on its face, hard.  I can't find a reference offhand, but I remember
> reading somewhere that CFQ is optimized for more desktop type
> workloads, and that in its efforts to ensure fair IO access for all
> processes, it can actively interfere with high-concurrency workloads
> like you'd expect to see on a DB server -- especially one as big as
> your specs indicate.  Then again, it's been a few years, so the
> scheduler may have improved significantly in that span.
>
> My standard approach since has just been to use no-op.  We've shelled
> out enough money for a RAID controller, if not a SAN, so it seems
> silly to me not to defer to the hardware, and let it do its job.  With
> big caches, command queueing, and direct knowledge of how the data is
> laid out on the spindles, I'm hard-pressed to imagine a scenario where
> the kernel is going to be able to do a better job of IO prioritization
> than the controller.
>
> I'd absolutely recommend testing with pg, so you can get a feel for
> how it behaves under real-world workloads.  The critical thing there
> is that your testing needs to create workloads that are in the
> neighborhood of what you'll see in production.  In my case, the final
> round of testing included something like 15-20% of the user-base for
> the app the db served, and everything seemed fine.  Once we opened the
> flood-gates, and all the users were hitting the new db, though,
> nothing worked for anyone.  Minute-plus page-loads across the board,
> when people weren't simply timing out.
>
> As always, YMMV, the plural of anecdote isn't data, &c.
>
> rls
>
> --
> :wq

I have a somewhat similar story. :)

We recently upgraded to RHEL 6 (2.6.32 + patches) from RHEL 5.6.

Our machines are:

24 core (4x6) X5670 2.93GHz
144Gb of RAM
2 x RAID 1 SAS - WAL (on a 5405Z)
8 x RAID10 SAS - Data (on a 5805Z)

We decided to test CFQ again (after using the deadline scheduler) and
it looked good in normal file system testing and what not.

Once we ramped up production traffic on the machines, PostgreSQL
pretty much died under the load and could never get to a steady state.
I think this had something to do with the PG backends not having
enough I/O bandwidth (due to CFQ) to put data into cache fast enough.
This went on for an hour before we decided to switch back to deadline.
The system was back to normal working order (with 5-6x the I/O
throughput of CFQ) in about 3 minutes, after which I/O wait was down
to 0-1%.

We run a (typical?) OLTP workload for a web app and see something like
2000 to 5000 req/s against PG.

Not sure if this helps in the OP's situation, but I guess it's one of
those things you need to test with a production workload to find out.
:)

Regards,
Omar