Re: Use durable_unlink for .ready and .done files for WAL segment removal

At Fri, 02 Nov 2018 14:47:08 +0000, Nathan Bossart <bossartn(at)amazon(dot)com> wrote in <154117002849(dot)5569(dot)14588306221618961668(dot)pgcf(at)coridan(dot)postgresql(dot)org>
> One argument for instead checking WAL file existence before calling
> archive_command might be to avoid the increased startup time.
> Granted, any added delay from this patch is unlikely to be noticeable
> unless your archiver is way behind and archive_status has a huge
> number of files. However, I have seen cases where startup is stuck on
> other tasks like SyncDataDirectory() and RemovePgTempFiles() for a
> very long time, so perhaps it is worth considering.

While archive_mode is tuned on, .ready files are created for all
exising wal files if not exists. Thus archiver may wait for the
ealiest segment to have .ready file. As the result
pgarch_readyXLog can be modified to loops over wal files, not
status files. This prevents the confusion comes from .ready
files for non-existent segment files.

RemoveXlogFile as is doesn't get confused by .done files for
nonexistent segments.

We may leave useless .done/.ready files. We no longer scan over
the files so no matter how many files are there in the directory.

The remaining issue is removal of the files. Even if we blew away
the directory altogether, status files would be cleanly recreated
having already-archived wal segments are archived again. However,
redundant copy won't happen with our recommending configuration:p

# Yeah, I see almost all sites uses simple 'cp' or 'scp' for that..

regards.

--
Kyotaro Horiguchi
NTT Open Source Software Center