Re: Streaming replication and WAL archive interactions

On Wed, Apr 22, 2015 at 6:42 AM, Robert Haas <robertmhaas(at)gmail(dot)com> wrote:
>
> On Tue, Apr 21, 2015 at 6:55 AM, Heikki Linnakangas <hlinnaka(at)iki(dot)fi> wrote:
> > On 04/21/2015 12:04 PM, Michael Paquier wrote:
> >> On Tue, Apr 21, 2015 at 4:38 PM, Heikki Linnakangas <hlinnaka(at)iki(dot)fi>
> >> wrote:
> >>> Note that even though we don't archive the partial last segment on the
> >>> previous timeline, the same WAL is copied to the first segment on the new
> >>> timeline. So the WAL isn't lost.
> >>
> >> But if the failed master has archived those segments safely, we may need
> >> them, no? I am not sure we can ignore a user who would want to do a PITR
> >> with recovery_target_timeline pointing to the one of the failed master.
> >
> > I think it would be acceptable. If you want to maintain an up-to-the-second
> > archive, you can use pg_receivexlog. Mind you, if the standby wasn't
> > promoted, the partial segment would not be present in the archive anyway.
> > And you can copy the WAL segment manually from 0000000200000000000000XX to
> > pg_xlog/0000000100000000000000XX before starting PITR.
> >
> > Another thought is that we could archive the partial file, but with a
> > different name to avoid confusing it with the full segment. For example, we
> > could archive a partial 000000010000000000000012 segment as
> > "000000020000000000000012.00000128.partial", where 00000128 indicates how
> > far that file is valid (this naming is similar to how the backup history
> > files are named). Recovery wouldn't automatically pick up those files, but
> > the DBA could easily copy the partial file into pg_xlog with the full
> > segment's name, if he wants to do PITR to that piece of WAL.
>
> So, suppose you A replicating to B (via an archive) replicating to C
> (via a separate archive); A dies, B is promoted. It sounds to me like
> today this will work and with your proposed change it will require
> manual intervention. I don't think that's OK.

This is going to change a behavior that people are used to for a
couple of releases. I would not mind having this patch do
"archive_mode = on during recovery" => archive only segments generated
by this node + the last partial segment on the old timeline at
promotion, without renaming to preserve backward compatible behavior.
If master and standby point to separate archive locations, this way
the operator can sort out later the one he would want to use. If they
point to the same location, archive_command scripts already do
internally such renaming, at least that's what I suspect an
experienced user would do, now it is true that not many people are
experienced in this area I see mistakes regarding such things on a
weekly basis... This .partial segment renaming is something that we
should let the archive_command manage with its internal logic.
Regards,
--
Michael