Re: Switching timeline over streaming replication

On 20.11.2012 15:33, Amit Kapila wrote:
> Defect-2:
> 1. start primary A
> 2. start standby B following A
> 3. start cascade standby C following B.
> 4. Start another standby D following C.
> 5. Execute the following commands in the primary A.
> create table tbl(f int);
> insert into tbl values(generate_series(1,1000));
> 6. Promote standby B.
> 7. Execute the following commands in the primary B.
> insert into tbl values(generate_series(1001,2000));
> insert into tbl values(generate_series(2001,3000));
>
> The following logs are observed on standby C:
>
> LOG: restarted WAL streaming at position 0/7000000 on tli 2
> ERROR: requested WAL segment 000000020000000000000007 has already been
> removed
> LOG: record with zero length at 0/7028190
> LOG: record with zero length at 0/7048540
> LOG: out-of-sequence timeline ID 1 (after 2) in log segment
> 000000020000000000000007, offset 0

Hmm, this one is actually a pre-existing bug. There's a sanity check
that the sequence of timeline IDs that are seen in the XLOG page headers
doesn't go backwards. In other words, if the last XLOG page that was
read had timeline id X, the next page must have a tli >= X. The startup
process keeps track of the last seen timeline id in lastPageTLI. In
standby_mode, when the startup process is reading from a pre-existing
file in pg_xlog (typically put there by streaming replication) and it
reaches the end of valid WAL (marked by an error in decoding it, ie.
"record with zero length" in your case), it sleeps for five seconds and
retries. At retry, the WAL file is re-opened, and as part of sanity
checking it, the first page header in the file is validated.

Now, if there was a timeline change in the current WAL segment, and
we've already replayed past that point, lastPageTLI will already be set
to the new TLI, but the first page on the file contains the old TLI.
When the file is re-opened, and the first page is validated, you get the
error.

The fix is quite straightforward: we should refrain from checking the
TLI when we re-open a WAL file. Or better yet, compare it against the
TLI we saw at the beginning of the last WAL segment, not the last WAL page.

I propose the attached patch (against 9.2) to fix that. This should be
backpatched to 9.0, where standby_mode was introduced. The code was the
same in 8.4, too, but AFAICS there was no problem there because 8.4
never tried to re-open the same WAL segment after replaying some of it.

- Heikki