Re: Why we are going to have to go DirectIO

Jonathan,

> For those interested in the details... (1) It's not quite 50/50, that's one
> bound for how the balance is allowed to go. (2) Anybody trying to add
> tunables to the kernel tends to run into resistance. Exposing thousands of
> knobs tends to lead to a situation where you *have* to be an expert on all
> those knobs to get decent behavior out of your system. So there is a big
> emphasis on having the kernel tune itself whenever possible. Here is a
> situation where that is not always happening, but a fix (which introduces
> no knob) is in the works.

Yeah, we get into this argument all the time. The problem is when you
run into situations where there is no optimal (or even acceptable)
setting for all, or even most, users. And I'll say in advance that 2Q
is one of those situations.

> As an example, I've never done much with the PostgreSQL knobs on the LWN
> server. I just don't have the time to mess with it, and things Work Well
> Enough.

Sure, and even when I teach fiddling with the knobs, there's only 12-20
knobs 95% of users need to have any interest in. But we have ~~220
settings for the other 5%, and those users would be screwed without them.

> Bugs and regressions happen, and I won't say that we do a good enough job
> in that regard. There has been some concern recently that we're accepting
> too much marginal stuff. We have problems getting enough people to
> adequately review code — I think I've heard of another project or two with
> similar issues :). But nobody sees the kernel as experimental or feels
> that the introduction of bugs is an acceptable thing.

OK. The chain of events over the pdflush bug really felt like what I
said earlier, especially since problems *were* reported shortly after
kernel release and ignored.

> I think you're talking to the wrong people.

Quite possibly.

> Perhaps even better: the next filesystem, storage, and memory management
> summit is March 24-25.

Link? I can't find anything Googling by that name. I'm pretty sure we
can get at least one person there.

> Gee, if only there were a web site where one could read about changes to
> the Linux kernel :)

Even you don't cover 100% of performance-changing commits. And I'll
admit to missing issues of LWN when I'm travelling.

> Seriously, though, one of the best things to do would be to make a point of
> picking up a kernel around -rc3 (right around now, say, for 3.13) and
> running a few benchmarks on it. If you report a performance regression at
> that stage, it will get attention.

Yeah, back to the "we need resources for good benchmarks" discussion
fork ...

--
Josh Berkus
PostgreSQL Experts Inc.
http://pgexperts.com