Re: Missing WAL file after running pg_rewind

On Thu, Jan 11, 2018 at 04:58:02PM +0000, Dylan Luong wrote:
> The steps I took were:
>
> 1. Stop all watchdogs
>
> 2. Start/stop the old master
>
> 3. Run 'checkpoint' on new master
>
> 4. Run the pg_rewind on old master to resync with new master
>
> 5. Start the old master (as new slave)

That's a sane flow to me.

> 2018-01-11 23:21:59 ACDT [112235]: [2-1] db=,user= app=,host= FATAL: could not receive data from WAL stre
> am: ERROR: requested WAL segment 0000000600000383000000BE has already been removed
>
> Has anyone experience this before with pg_rewind?

When restarting a standby after a rewind has been done to it, note that,
in order to recover to a consistent point, it needs to replay WAL from
the previous checkpoint checkpoint where WAL has forked during the
promotion up to the point where the rewind has finished. Per your logs,
I am getting that the previous checkpoint before the timeline jump is
located in segment 0000000X00000383000000BE, but this did not get
archived.

> The earliest wall files in the archive directory was around just after the failover occurred.
>
> Eg, in the archive directory on the new Master:
> $ ls -l
> total 15745032
> -rw-------. 1 postgres postgres 16777216 Jan 11 17:52 0000000500000383000000C0.partial
> -rw-------. 1 postgres postgres 16777216 Jan 11 17:52 0000000600000383000000C0
> -rw-------. 1 postgres postgres 16777216 Jan 11 17:52 0000000600000383000000C1
> -rw-------. 1 postgres postgres 16777216 Jan 11 17:52 0000000600000383000000C2

Yeah, you are looking for the WAL segment just before the last, partial
WAL segment of the previous timeline. Depending on your archiving
strategy, I guess that you should have set archive_mode = 'always' so as
the server which was the standby before the promotion is also able to
store them.
--
Michael