Re: Use of O_DIRECT only for open_* sync options

Greg Smith wrote:
> Bruce Momjian wrote:
> > xlogdefs.h says:
> >
> > /*
> > * Because O_DIRECT bypasses the kernel buffers, and because we never
> > * read those buffers except during crash recovery, it is a win to use
> > * it in all cases where we sync on each write(). We could allow O_DIRECT
> > * with fsync(), but because skipping the kernel buffer forces writes out
> > * quickly, it seems best just to use it for O_SYNC. It is hard to imagine
> > * how fsync() could be a win for O_DIRECT compared to O_SYNC and O_DIRECT.
> > * Also, O_DIRECT is never enough to force data to the drives, it merely
> > * tries to bypass the kernel cache, so we still need O_SYNC or fsync().
> > */
> >
> > This seems wrong because fsync() can win if there are two writes before
> > the sync call. Can kernels not issue fsync() if the write was O_DIRECT?
> > If that is the cause, we should document it.
> >
>
> The comment does look busted, because you did imagine exactly a case
> where they might be combined. The only incompatibility that I'm aware
> of is that O_DIRECT requires reads and writes to be aligned properly, so
> you can't use it in random application code unless it's aware of that.
> O_DIRECT and fsync are compatible; for example, MySQL allows combining
> the two: http://dev.mysql.com/doc/refman/5.1/en/innodb-parameters.html
>
> (That whole bit of documentation around innodb_flush_method includes
> some very interesting observations around O_DIRECT actually)
>
> I'm starting to consider the idea that much of the performance gains
> seen on earlier systems with O_DIRECT was because it decreased CPU usage
> shuffling things into the OS cache, rather than its impact on avoiding
> pollution of said cache. On Linux for example, its main accomplishment
> is decribed like this: "File I/O is done directly to/from user space
> buffers."
> http://www.kernel.org/doc/man-pages/online/pages/man2/open.2.html The
> earliest paper on the implementation suggests a big decrease in CPU
> overhead from that:
> http://www.ukuug.org/events/linux2001/papers/html/AArcangeli-o_direct.html
>
> Impossible to guess whether that's more true ("CPU cache pollution is a
> bigger problem now") or less true ("drives are much slower relative to
> CPUs now") today. I'm trying to remain agnostic and let the benchmarks
> offer an opinion instead.

Agreed. Perhaps we need a separate setting to turn direct I/O on and
off, and decouple wal_sync_method and direct I/O.

--
Bruce Momjian <bruce(at)momjian(dot)us> http://momjian.us
EnterpriseDB http://enterprisedb.com

+ It's impossible for everything to be true. +