Re: io_uring support

Hi,

On 2019-08-19 20:20:46 +0200, Dmitry Dolgov wrote:
> For already some time I'm following the new linux IO interface "io_uring", that
> was introduced relatively recently [1]. Short description says:
>
> Shared application/kernel submission and completion ring pairs, for
> supporting fast/efficient IO.

Yes, it's quite promising. I also played around some with it. One
thing I particularly like is that it seems somewhat realistic to have an
abstraction that both supports io_uring and window's iocp - personally I
don't think we need support for more than those.

> For us the important part is probably that it's an asynchronious IO, that can
> work not only with O_DIRECT, but with also with buffered access.

Note that while the buffered access does allow for some acceleration, it
currently does have quite noticable CPU overhead.

> Since I haven't found that many discussions in the hackers archives about async
> IO, and out of curiosity decided to prepare an experimental patch to see how
> this would looks like to use io_uring in PostgreSQL.

Cool!

> I've tested this patch so far only inside a qemu vm on the latest
> io_uring branch from linux-block tree. The result is relatively
> simple, and introduces new interface smgrqueueread, smgrsubmitread and
> smgrwaitread to queue any read we want, then submit a queue to a
> kernel and then wait for a result. The simplest example of how this
> interface could be used I found in pg_prewarm for buffers prefetching.

Hm. I'm bit doubtful that that's going in the direction of being the
right interface. I think we'd basically have to insist that all AIO
capable smgr's use one common AIO layer (note that the UNDO patches add
another smgr implementation). Otherwise I think we'll have a very hard
time to make them cooperate. An interface like this would also lead to
a lot of duplicated interfaces, because we'd basically need most of the
smgr interface functions duplicated.

I suspect we'd rather have to build something where the existing
functions grow a parameter controlling synchronizity. If AIO is allowed
and supported, the smgr implementation would initiate the IO, together
with a completion function for it, and return some value allowing the
caller to wait for the result if desirable.

> As a result of this experiment I have few questions, open points and requests
> for the community experience:
>
> * I guess the proper implementation to use async IO is a big deal, but could
> bring also significant performance advantages. Is there any (nearest) future
> for such kind of async IO in PostgreSQL? Buffer prefetching is a simplest
> example, but taking into account that io_uring supports ordering, barriers
> and linked events, there are probably more use cases when it could be useful.

The lowest hanging fruit that I can see - and which I played with - is
making the writeback flushing use async IO. That's particularly
interesting for bgwriter. As it commonly only performs random IO, and
as we need to keep the number of dirty buffers in the kernel small to
avoid huge latency spikes, being able to submit IOs asynchronously can
yield significant benefits.

> * Assuming that the answer for previous question is positive, there could be
> different strategies how to use io_uring. So far I see different
> opportunities for waiting. Let's say we have prepared a batch of async IO
> operations and submitted it. Then we can e.g.
>
> -> just wait for a batch to be finished
> -> wait (in the same syscall as submitting) for previously submitted batches,
> then start submitting again, and at the end wait for the leftovers
> -> peek if there are any events completed, and get only those without waiting
> for the whole batch (in this case it's necessary to make sure submission
> queue is not overflowed)
>
> So it's open what and when to use.

I don't think there's much point in working only with complete
batches. I think we'd loose too much of the benefit by introducing
unnecessary synchronous operations. I think we'd need to design the
interface in a way that there constantly can be in-progress IOs, block
when the queue is full, and handle finished IOs using a callback
mechanism or such.

> * Does it makes sense to use io_uring for smgrprefetch? Originally I've added
> io_uring parts into FilePrefetch also (in the form of preparing and submiting
> just one buffer), but not sure if this API is suitable.

I have a hard time seeing that being worthwhile, unless we change the
way it's used significantly. I think to benefit really, we'd have to be
able to lock multiple buffers, and have io_uring prefetch directly into
buffers.

> * How may look like a data structure, that can describe IO from PostgreSQL
> perspective? With io_uring we need to somehow identify IO operations that
> were completed. For now I'm just using a buffer number.

In my hacks I've used the sqe's user_data to point to a struct with
information about the IO.

> Btw, this
> experimental patch has many limitations, e.g. only one ring is used for
> everything, which is of course far from ideal and makes identification even
> more important.

I think we don't want to use more than one ring. Makes it too
complicated to have interdependencies between operations (e.g. waiting
for fsyncs before submitting further writes). I also don't really see
why we would benefit from more?

> * There are few more freedom dimensions, that io_uring introduces - how many
> rings to use, how many events per ring (which is going to be n for sqe and
> 2*n for cqe), how many IO operations per event to do (similar to
> preadv/pwritev we can provide a vector), what would be the balance between
> submit and complete queues. I guess it will require a lot of benchmarking to
> find a good values for these.

One thing you didn't mention: A lot of this also requires that we
overhaul the way buffer locking for IOs works. Currently we really can
only have one proper IO in progress at a time, which clearly isn't
sufficient for anything that wants to use AIO.

Greetings,

Andres Freund