Re: .ready and .done files considered harmful

+ /*
+ * Perform a full directory scan to identify the next log segment. There
+ * may be one of the following scenarios which may require us to perform a
+ * full directory scan.
+ *
+ * 1. This is the first cycle since archiver has started and there is no
+ * idea about the next anticipated log segment.
+ *
+ * 2. There is a timeline switch, i.e. the timeline ID tracked at archiver
+ * does not match with current timeline ID. Archive history file as part of
+ * this timeline switch.
+ *
+ * 3. The next anticipated log segment is not available.

One benefit of the current implementation of pgarch_readyXlog() is
that .ready files created out of order will be prioritized before
segments with greater LSNs. IIUC, with this patch, as long as there
is a "next anticipated" segment available, the archiver won't go back
and archive segments it missed. I don't think the archive status
files are regularly created out of order, but XLogArchiveCheckDone()
has handling for that case, and the work to avoid creating .ready
files too early [0] seems to make it more likely. Perhaps we should
also force a directory scan when we detect that we are creating a
.ready file for a segment that is older than the "next anticipated"
segment.

Nathan

[0] https://postgr.es/m/DA71434B-7340-4984-9B91-F085BC47A778%40amazon.com