Re: Tracking down log segment corruption

On Sun, May 2, 2010 at 12:10 PM, Tom Lane <tgl(at)sss(dot)pgh(dot)pa(dot)us> wrote:

> No, this would be a pg_database row with that OID. But it looks like
> you found the relevant index anyway.
>
> Yup, realized that on second reading.

> > These commands worked fine on the master, yet this seems suspiciously
> > relevant.
>
>
> Yeah, perhaps so. What time did the failure on the standby occur (how
> long after you did those moves)? Is it reasonable to assume that this
> was the first subsequent access to the index?
>
>
Bingo. Yes it is reasonable. It was 25 seconds between my altering the
index in question and the server crash.

My local commands (in MDT, plus my machine is 15 sec ahead of the server):

09:10:52> alter index cts_20100501_natural_uk set tablespace ts30;
ALTER INDEX
Time: 787.790 ms
09:11:41> alter index cts_20100501_pkey set tablespace ts30;
ALTER INDEX
Time: 468.526 ms
09:11:51> alter index cts_20100501_topic_date_nk set tablespace ts30;
ALTER INDEX
Time: 385.322 ms
09:11:59> alter index cts_20100501_updated_nk set tablespace ts30;
ALTER INDEX
Time: 963.150 ms
09:12:10> alter table cts_20100501 set tablespace ts29;
ALTER TABLE

And from the wsb log (times in EDT):

4158 2010-05-02 11:12:09 EDT [26446]LOG: restored log file
"0000000100003C77000000C4" from archive
4158 2010-05-02 11:12:09 EDT [26447]WARNING: specified item offset is too
large
4158 2010-05-02 11:12:09 EDT [26448]CONTEXT: xlog redo insert: rel
48777166/22362/48778276; tid 2/2
4158 2010-05-02 11:12:09 EDT [26449]PANIC: btree_insert_redo: failed to
add item