Re: Re: [Lsf-pc] Linux kernel impact on PostgreSQL performance (summary v2 2014-1-17)

Hi Mel,

On 2014-01-17 16:31:48 +0000, Mel Gorman wrote:
> Direct IO, buffered IO, double buffering and wishlists
> ------------------------------------------------------
> 3. Hint that a page should be dropped immediately when IO completes.
> There is already something like this buried in the kernel internals
> and sometimes called "immediate reclaim" which comes into play when
> pages are bgin invalidated. It should just be a case of investigating
> if that is visible to userspace, if not why not and do it in a
> semi-sensible fashion.

"bgin invalidated"?

Generally, +1 on the capability to achieve such a behaviour from
userspace.

> 7. Allow userspace process to insert data into the kernel page cache
> without marking the page dirty. This would allow the application
> to request that the OS use the application copy of data as page
> cache if it does not have a copy already. The difficulty here
> is that the application has no way of knowing if something else
> has altered the underlying file in the meantime via something like
> direct IO. Granted, such activity has probably corrupted the database
> already but initial reactions are that this is not a safe interface
> and there are coherency concerns.

I was one of the people suggesting that capability in this thread (after
pondering about it on the back on my mind for quite some time), and I
first though it would never be acceptable for pretty much those
reasons.
But on second thought I don't think that line of argument makes too much
sense. If such an API would require write permissions on the file -
which it surely would - it wouldn't allow an application to do anything
it previously wasn't able to.
And I don't see the dangers of concurrent direct IO as anything
new. Right now the page's contents reside in userspace memory and aren't
synced in any way with either the page cache or the actual on disk
state. And afaik there are already several data races if a file is
modified and read both via the page cache and direct io.

The scheme that'd allow us is the following:
When postgres reads a data page, it will continue to first look up the
page in its shared buffers, if it's not there, it will perform a page
cache backed read, but instruct that read to immediately remove from the
page cache afterwards (new API or, posix_fadvise() or whatever). As long
as it's in shared_buffers, postgres will not need to issue new reads, so
there's no no benefit keeping it in the page cache.
If the page is dirtied, it will be written out normally telling the
kernel to forget about the caching the page (using 3) or possibly direct
io).
When a page in postgres's buffers (which wouldn't be set to very large
values) isn't needed anymore and *not* dirty, it will seed the kernel
page cache with the current data.

Now, such a scheme wouldn't likely be zero-copy, but it would avoid
double buffering. I think the cost of buffer copying has been overstated
in this thread... he major advantage is that all that could easily
implemented in a very localized manner, without hurting other OSs and it
could easily degrade on kernels not providing that capability, which
would surely be the majority of installations for the next couple of
cases.

So, I think such an interface would be hugely beneficial - and I'd be
surprised if other applications couldn't reuse it. And I don't think
it'd be all that hard to implement on the kernel side?

> Dave Chinner asked "why, exactly, do you even need the kernel page
> cache here?" when Postgres already knows how and when data should
> be written back to disk. The answer boiled down to "To let kernel do
> the job that it is good at, namely managing the write-back of dirty
> buffers to disk and to manage (possible) read-ahead pages". Postgres
> has some ordering requirements but it does not want to be responsible
> for all cache replacement and IO scheduling. Hannu Krosing summarised
> it best as

The other part is that using the page cache for the majority of warm,
but not burning hot pages, allows the kernel to much more sensibly adapt
to concurrent workloads requiring memory in some form or other (possibly
giving it to other VMs when mostly idle and such).

> 8. Allow copy-on-write of page-cache pages to anonymous. This would limit
> the double ram usage to some extent. It's not as simple as having a
> MAP_PRIVATE mapping of a file-backed page because presumably they want
> this data in a shared buffer shared between Postgres processes. The
> implementation details of something like this are hairy because it's
> mmap()-like but not mmap() as it does not have the same writeback
> semantics due to the write ordering requirements Postgres has for
> database integrity.

> 9. Hint that a page in an anonymous buffer is a copy of a page cache
> page and invalidate the page cache page on COW. This limits the
> amount of double buffering. It's in as a low priority item as it's
> unclear if it's really necessary and also I suspect the implementation
> would be very heavy because of the amount of information we'd have
> to track in the kernel.
>

I don't see this kind of proposals going anywhere. The amounts of
changes to postgres and the kernel sound prohibitive to me, besides the
utter crummyiness.

Greetings,

Andres Freund