Re: AIO writes vs hint bits vs checksums

Hi,

Attached is a, unfortunately long, series of patches implementing what I
described upthread.

0001 Add very basic test for kill_prior_tuples

We currently don't exercise this patch for gist and hash, which seems
somewhat criminal.

0002 heapam: Move logic to handle HEAP_MOVED into a helper function

A prep patch, which consolidates HEAP_MOVED handling into a helper. This
isn't strictly necessary, but I got bothered by it while writing this patch
series.

0003 bufmgr: Add BufferLockHeldByMe()

Useful for writing assertions in later patches.

0004 heapam: Use exclusive lock on old page in CLUSTER

When I started looking into this I wanted to make sure that it's actually
safe to skip setting hint bits in all places. One place that's *not* safe
is cluster - we assume that all hint bits are set in rewrite_heap_tuple().

This might not strictly be required, but it seemed very fragile as
is. While using an exclusive lock does seem like it makes it a bit safer,
it's not a very good fix, as we still dirty the old buffer, which seems
like a shame when one uses VACUUM FULL.

0005 heapam: Only set tuple's block once per page in pagemode

Minor, but measurable, optimization.

0006 heapam: Add batch mode mvcc check and use it in page mode

This is a good bit faster on its own, but is required to avoid a
performance regression later, when setting hint bits only only when no IO
going on at the same.

0007 bufmgr: Make it easier to change number of buffer state bits

This just makes it easier to change the division of bits in
BufferDesc->state.

0008 bufmgr: Add interface to acquire right to set hint bits

The main change. Adds BufferPrepareToSetHintBits() which, if true is
returned, gives the rigth to modify a buffer without an exclusive
lock. Once the modification is done BufferFinishSetHintBits() needs to be
called.

I dithered a bit on the name, we don't really call them hint bits in all
the places that modify buffers without an exlusive lock.

The new infra is unused as of this commit.

0009 heapam: Acquire right to set hint bits

Moves over heapam to the new interface.

0010 Acquire right to set hint bits in the remaining places

Moves indexams and freespace/ over to the new interface.

0011 bufmgr: Detect some missing BufferPrepareToSetHintBits() calls

Now that we don't set hint bits without calling
BufferPrepareToSetHintBits() first, we can detect many missing calls by
adding an assertion to MarkBufferDirtyHint().

0012 bufmgr: Don't copy pages while writing out

Finally remove the copy of the buffer while writing it out.

0013 WIP: bufmgr: Detect some bad buffer accesses

This is a lot more thorough infrastructure for detecting modifications of
buffers without appropriate locks, by using mprotect() to detect such
cases. Only works when not using huge pages, and not on windows (not sure
how mprotect() is spelled there) and not arm macos (due to 16kB pages).

Has two modes, one just disallows modifications, and is reasonably
cheap. The more expensive mode disallows any access when the buffer is
unpinned. The commit message has some performance numbers.

This has proven to be a good investment as far as this thread goes - I
wasn't initially aware that freespace modified buffers without a lock and
this quickly found that.

Greetings,

Andres Freund