Re: Failback to old master

On 11/12/2014 03:28 PM, Ants Aasma wrote:
> On Tue, Nov 11, 2014 at 11:52 PM, Maeldron T. <maeldron(at)gmail(dot)com> wrote:
>> As far as I remember (I can’t test it right now but I am 99% sure) promoting the slave makes it impossible to connect the old master to the new one without making a base_backup. The reason is the timeline change. It complains.
>
> A safely shut down master (-m fast is safe) can be safely restarted as
> a slave to the newly promoted master. Fast shutdown shuts down all
> normal connections, does a shutdown checkpoint and then waits for this
> checkpoint to be replicated to all active streaming clients.

Right. You have to be careful to make sure the standby really did fully
catch up with the master, though. If it happens that the replication
connection is momentarily down when you shut down the master, for
example, then the master won't wait for the standby. You can use
pg_controlinfo to verify that, before promoting the standby.

> In case of a unsafe shut down (crash) it is possible that you have WAL
> lying around that was not streamed out to the slave.

Right.

> In this case the
> old master will request recovery from a point after the timeline
> switch and the new master will reply with an error. So it is safe to
> try re-adding a crashed master as a slave, but this might fail.

Are you sure it's guaranteed to fail, when the master had some WAL that
was not streamed before the crash? I'm not 100% sure about that. I
thought the old master might continue streaming and replaying, if there
just happens to be a record start at the same point in WAL on both
timelines. I think you'll get an error at the next checkpoint record,
because its timeline ID isn't what the old master expects, but it
would've started up already.

> Success is more likely when the whole operating system went down, as
> then it's somewhat likely that any WAL got streamed out before it made
> it to disk.
>
> In general my suggestion is to avoid slave promotion by removal of
> recovery.conf, it's too easy to get confused and end up with hard to
> diagnose data corruption.

Yeah, you can't emphasize that too much. Never remove recovery.conf.
That removes all the safeguards, and it's very easy to get a database
that looks OK at first glance, but is in fact corrupt.

- Heikki