Re: refactoring relation extension and BufferAlloc(), faster COPY

Hi,

Attached is v6. Changes:

- Try to address Melanie and Horiguchi-san's review. I think there's one or
two further things that need to be done

- Avoided inserting newly extended pages into the FSM while holding a buffer
lock. If we need to do so, we now drop the buffer lock and recheck if there
still is space (very very likely). See also [1]. I use the infrastructure
introduced over in that in this patchset.

- Lots of comment and commit message polishing. More needed, particularly for
the latter, but ...

- Added a patch to fix the pre-existing undefined behaviour in localbuf.c that
Melanie pointed out. Plan to commit that soon.

- Added a patch to fix some pre-existing DO_DB() format code issues. Plan to
commit that soon.

I did some benchmarking on "bufmgr: Acquire and clean victim buffer
separately" in isolation. For workloads that do a *lot* of reads, that proves
to be a substantial benefit on its own. For the, obviously unrealistically
extreme, workload of N backends doing
SELECT pg_prewarm('pgbench_accounts', 'buffer');
in a scale 100 database (with a 1281MB pgbench_accounts) and shared_buffers of
128MB, I see > 2x gains at 128, 512 clients. Of course realistic workloads
will have much smaller gains, but it's still good to see.

Looking at the patchset, I am mostly happy with the breakdown into individual
commits. However "bufmgr: Move relation extension handling into
ExtendBufferedRel{By,To,}" is quite large. But I don't quite see how to break
it into smaller pieces without making things awkward (e.g. due to static
functions being unused, or temporarily duplicating the code doing relation
extensions).

Greetings,

Andres Freund

[1] https://www.postgresql.org/message-id/20230325025740.wzvchp2kromw4zqz%40awork3.anarazel.de