Re: archive status ".ready" files may be created too early

Hello.

At Wed, 18 Dec 2019 14:10:04 +0900, Fujii Masao <masao(dot)fujii(at)gmail(dot)com> wrote in
> On Fri, Dec 13, 2019 at 7:50 AM Bossart, Nathan <bossartn(at)amazon(dot)com> wrote:
> >
> > Hi hackers,
> >
> > I believe I've uncovered a bug that may cause archive status ".ready"
> > files to be created too early, which in turn may cause an incorrect
> > version of the corresponding WAL segment to be archived.
> >
> > The crux of the issue seems to be that XLogWrite() does not wait for
> > the entire record to be written to disk before creating the ".ready"
> > file. Instead, it just waits for the last page of the segment to be
> > written before notifying the archiver. If PostgreSQL crashes before
> > it is able to write the rest of the record, it will end up reusing the
> > ".ready" segment at the end of crash recovery. In the meantime, the
> > archiver process may have already processed the old version of the
> > segment.
>
> Maybe I'm missing something... But since XLogWrite() seems to
> call issue_xlog_fsync() before XLogArchiveNotifySeg(), ISTM that
> this trouble shouldn't happen. No?

The trouble happens after the synced file is archived. If the last
record in the arcvhied segment was the first half of a continuation
record and crash before writing the last half, crash recovery stops
just before the first half and different record can be overwitten. As
the result the archived version of the segment becomes rotten.

regards.

--
Kyotaro Horiguchi
NTT Open Source Software Center