Re: Asynchronous and "direct" IO support for PostgreSQL.

> 2021年9月1日 13:56，Andres Freund <andres(at)anarazel(dot)de> 写道：
>
> Hi,
>
> Attached is an updated patch AIO series. The major changes are:
> - rebased onto master (Andres)
> - lots of progress on posix AIO backend (Thomas)
> - lots of progress towards a windows native AIO implementation - not yet quite
> merged (Thomas & David)
> - considerably improved "worker" io_method (Thomas)
> - some preliminary patches merged (Thomas) and thus dropped
> - error handling overhaul, AIO references now use resource owners
> - quite a few more localized bugfixes
> - further CI improvements
>
> Unfortunately there's a few tests that don't pass on windows. At least some of
> those failures also happen on master - hence the alternative output file added
> in the last commit.
>
> Thanks to Thomas there's now a new wiki page for AIO support:
> https://wiki.postgresql.org/wiki/AIO
> It's currently mostly a shared todo list....
>
> My own next steps are to try to get some of the preliminary patches merged
> into master, and to address some of the higher level things that aren't yet
> quite right with the AIO interface, and to split the "main" AIO patch into
> smaller patches.
>
> I hope that we soon send in a new version with native AIO support for
> windows. I'm mostly interested in that to make sure that we get the shared
> infrastructure right.
>
> Melanie has some work improving bitmap heap scan AIO support and some IO stats
> / explain improvements.
>
> I think a decent and reasonably simple example for the way the AIO interface
> can be used to do faster IO is
> v3-0028-aio-Use-AIO-in-nbtree-vacuum-scan.patch.gz which adds AIO for nbtree
> vacuum. It's not perfectly polished, but I think it shows that it's not too
> hard to add AIO usage to individual once the general infrastructure is in
> place.
>
> I've attached the code for posterity, but the series is large enough that I
> don't think it makes sense to do that all that often... The code is at
> https://github.com/anarazel/postgres/tree/aio

HI Andres:

I noticed this feature and did some testing.
code in GitHub's aio branch:
Function
static void
pgaio_write_smgr_retry(PgAioInProgress *io)
{
uint32 off;
AioBufferTag *tag = &io->scb_data.write_smgr.tag;
SMgrRelation reln = smgropen(tag->rlocator.locator, tag->rlocator.backend);

io->op_data.read.fd = smgrfd(reln, tag->forkNum, tag->blockNum, &off);
Assert(off == io->op_data.read.offset);
}

seems should to be:
io->op_data.write.fd = smgrfd(reln, tag->forkNum, tag->blockNum, &off);
Assert(off == io->op_data.write.offset);

Best regards,
Wenjing

>
> Greetings,
>
> Andres Freund
> <v3-0001-windows-Only-consider-us-to-be-running-as-service.patch.gz><v3-0002-WIP-Fix-non-aio-bug-leading-to-checkpointer-not-s.patch.gz><v3-0003-aio-WIP-align-PGAlignedBlock-to-page-size.patch.gz><v3-0004-Add-allocator-support-for-larger-allocation-align.patch.gz><v3-0005-ilist.h-debugging-improvements.patch.gz><v3-0006-lwlock-xlog-Report-caller-wait-event-for-LWLockWa.patch.gz><v3-0007-heapam-Don-t-re-inquire-block-number-for-each-tup.patch.gz><v3-0008-Add-pg_prefetch_mem-macro-to-load-cache-lines.patch.gz><v3-0009-heapam-WIP-cacheline-prefetching-for-hot-pruning.patch.gz><v3-0010-WIP-Change-instr_time-to-just-store-nanoseconds-t.patch.gz><v3-0011-aio-Add-some-error-checking-around-pinning.patch.gz><v3-0012-aio-allow-lwlocks-to-be-unowned.patch.gz><v3-0013-condvar-add-ConditionVariableCancelSleepEx.patch.gz><v3-0014-Use-a-global-barrier-to-fix-DROP-TABLESPACE-on-Wi.patch.gz><v3-0015-pg_buffercache-Add-pg_buffercache_stats.patch.gz><v3-0016-bufmgr-Add-LockBufHdr-fastpath.patch.gz><v3-0017-lwlock-WIP-add-extended-locking-functions.patch.gz><v3-0018-io-Add-O_DIRECT-non-buffered-IO-mode.patch.gz><v3-0019-io-Increase-default-ringbuffer-size.patch.gz><v3-0020-Use-aux-process-resource-owner-in-walsender.patch.gz><v3-0021-Ensure-a-resowner-exists-for-all-paths-that-may-p.patch.gz><v3-0022-aio-Add-asynchronous-IO-infrastructure.patch.gz><v3-0023-aio-Use-AIO-in-pg_prewarm.patch.gz><v3-0024-aio-Use-AIO-in-bulk-relation-extension.patch.gz><v3-0025-aio-Use-AIO-in-checkpointer-bgwriter.patch.gz><v3-0026-aio-bitmap-heap-scan-Minimal-and-hacky-improvemen.patch.gz><v3-0027-aio-Use-AIO-in-heap-vacuum-s-lazy_scan_heap-and-l.patch.gz><v3-0028-aio-Use-AIO-in-nbtree-vacuum-scan.patch.gz><v3-0029-aio-Use-AIO-for-heap-table-scans.patch.gz><v3-0030-aio-Use-AIO-in-SyncDataDirectory.patch.gz><v3-0031-aio-Use-AIO-in-ProcessSyncRequests.patch.gz><v3-0032-aio-wal-concurrent-WAL-flushes.patch.gz><v3-0033-wal-Use-LWLockAcquireOrWait-in-AdvanceXLInsertBuf.patch.gz><v3-0034-wip-wal-async-commit-reduce-frequency-of-latch-se.patch.gz><v3-0035-aio-wal-padding-of-partial-records.patch.gz><v3-0036-aio-wal-extend-pg_stat_wal.patch.gz><v3-0037-aio-initial-sketch-for-design-document.patch.gz><v3-0038-aio-CI-and-README.md.patch.gz><v3-0039-XXX-Add-temporary-workaround-for-partition_prune-.patch.gz>