Re: Pre-allocating WAL files

On 10/6/21, 9:34 AM, "Bossart, Nathan" <bossartn(at)amazon(dot)com> wrote:
> I have been thinking about the right place to put this logic. My
> first thought is that it sounds like something that ought to go in the
> WAL writer process, but as Andres noted upthread, that is undesirable
> because it'll add latency for asynchronous commits. The checkpointer
> process seems to be another candidate, but I'm not totally sure how
> it'll fit in. My patch works by maintaining a small pool of pre-
> allocated segments that is quickly replenished whenever one is used.
> If the checkpointer is spending most of its time checkpointing, this
> small pool could remain empty for long periods of time. To keep pre-
> allocating WAL while we're checkpointing, perhaps we could add another
> function like CheckpointWriteDelay() that is called periodically.
> There still might be several operations in CheckPointGuts() that take
> a while and leave the segment pool empty, but maybe that's okay for
> now.

Here is a first attempt at adding the pre-allocation logic to the
checkpointer. I went ahead and just used CheckpointWriteDelay() for
pre-allocating during checkpoints. I've done a few pgbench runs, and
it seems to work pretty well. Initialization is around 15% faster,
and I'm seeing about a 5% increase in TPS with a simple-update
workload with wal_recycle turned off. Of course, these improvements
go away once segments can be recycled.

Nathan