Re: .ready and .done files considered harmful

Greetings,

* Robert Haas (robertmhaas(at)gmail(dot)com) wrote:
> On Tue, May 4, 2021 at 11:54 AM Dilip Kumar <dilipbalaut(at)gmail(dot)com> wrote:
> > I agree that if we continue to archive one file using the archive
> > command then Robert's solution of checking the existence of the next
> > WAL segment (N+1) has an advantage. But, currently, if you notice
> > pgarch_readyXlog always consider any history file as the oldest file
> > but that will not be true if we try to predict the next WAL segment
> > name. For example, if we have archived 000000010000000000000004 then
> > next we will look for 000000010000000000000005 but after generating
> > segment 000000010000000000000005, if there is a timeline switch then
> > we will have the below files in the archive status
> > (000000010000000000000005.ready, 00000002.history file). Now, the
> > existing archiver will archive 00000002.history first whereas our code
> > will archive 000000010000000000000005 first. Said that I don't see
> > any problem with that because before archiving any segment file from
> > TL 2 we will definitely archive the 00000002.history file because we
> > will not find the 000000010000000000000006.ready and we will scan the
> > full directory and now we will find 00000002.history as oldest file.
>
> OK, that makes sense and is good to know.

I expect David will chime in on this thread too, but I did want to point
out that when it coming to archiving history files you'd *really* like
that to be done just about as quickly as absolutely possible, to avoid
the case that we saw before that code was added, to wit: two promotions
done too quickly that ended up with conflicting history and possibly
conflicting WAL files trying to be archived, and ensuing madness.

It's not just about making sure that we archive the history file for a
timeline before archiving WAL segments along that timeline but also
about making sure we get that history file into the archive as fast as
we can, and archiving a 16MB WAL first would certainly delay that.

Thanks,

Stephen