Re: [HACKERS] Clock with Adaptive Replacement

On Wed, Apr 25, 2018 at 6:31 PM, Andrey Borodin <x4mmm(at)yandex-team(dot)ru> wrote:
> 4. Using O_DIRECT while writing data files

One interesting thing about O_DIRECT that I haven't seen mentioned in
these discussions:

POSIX effectively requires writes to the same file to be serialised,
meaning in practice that [p]write[v]() acquires a mutex on the inode.
I've heard of people working on other databases that have multiple
write-back threads that considered this to be a serious problem, and I
guess it becomes worse as you concentrate more stuff into bigger files
to cut down on file descriptors and file system metadata operations
and have more concurrent writers. Some ways around that:

1. On Linux if you use O_DIRECT then XFS forgets about that POSIX
requirement (O_DIRECT is outside the spec anyway). It just doesn't
acquire inode->i_mutex, so you get parallel writes to the same file
(except in some corner cases). AFAIK other Linux filesystems don't do
that, so XFS might be your only choice if you want parallel direct IO
on that OS. Anyone know more about that? Relevant kernel code: the
inode_lock() call that appears in ext4_file_write_iter() but not in
xfs_file_write_iter() in the IOCB_DIRECT case.

2. Ditto for Solaris's UFS, AIX's JFS2 with "O_CIO" enabled (but
apparently not the JFS port on Linux). I don't know the answer for
FreeBSD's UFS off-hand.

3. I've heard that ZFS achieves parallel writes to the same file
while magically adhering to the POSIX serialisation rules, and that's
independent of direct IO (which it doesn't even support).

--
Thomas Munro
http://www.enterprisedb.com