Server Panic when trying to stop point in time recovery

From: Chris Copeland <chris(at)cope360(dot)com>
To: pgsql-general(at)postgresql(dot)org
Subject: Server Panic when trying to stop point in time recovery
Date: 2010-04-27 20:39:40
Message-ID: z2id0b407961004271339pa9533846h66772b5ac33c83b7@mail.gmail.com
Views: Raw Message | Whole Thread | Download mbox | Resend email
Thread:
Lists: pgsql-general

I am running 8.2.4 on Solaris 10 x86.

I have setup WAL file shipping from a primary server to a warm standby. I
am able to start the standby server using a backup from the primary and get
it to apply the log files as they arrive. My problem comes when I want to
trigger the standby server to come out of recovery mode.

Here is the log file starting from when the server comes up from the
backup. Just prior to the error I have "triggered" the server to exit
recovery mode by making my restore_command return 1 instead of 0.

2010-04-27 15:00:58 CDT :LOG: database system was interrupted at 2010-04-27
10:10:08 CDT
2010-04-27 15:00:58 CDT :LOG: starting archive recovery
2010-04-27 15:00:58 CDT :LOG: restore_command = "/opt/data/restore.sh
/opt/wal/archwalremote/%f %p"
2010-04-27 15:00:58 CDT :LOG: restored log file
"000000010000009F000000BA.00000278.backup" from archive
2010-04-27 15:00:59 CDT :LOG: restored log file "000000010000009F000000BA"
from archive
2010-04-27 15:00:59 CDT :LOG: checkpoint record is at 9F/BA000278
2010-04-27 15:00:59 CDT :LOG: redo record is at 9F/BA000278; undo record is
at 0/0; shutdown FALSE
2010-04-27 15:00:59 CDT :LOG: next transaction ID: 0/325985316; next OID:
823081
2010-04-27 15:00:59 CDT :LOG: next MultiXactId: 2127; next MultiXactOffset:
4278
2010-04-27 15:00:59 CDT :LOG: automatic recovery in progress
2010-04-27 15:00:59 CDT :LOG: redo starts at 9F/BA0002C0
2010-04-27 15:01:00 CDT :LOG: restored log file "000000010000009F000000BB"
from archive
2010-04-27 15:01:02 CDT :LOG: restored log file "000000010000009F000000BC"
from archive

<snip, many more files restored>

2010-04-27 15:03:19 CDT :LOG: restored log file "000000010000009F000000FE"
from archive
2010-04-27 15:03:20 CDT :LOG: restored log file "00000001000000A000000000"
from archive
2010-04-27 15:06:00 CDT :LOG: restored log file "00000001000000A000000001"
from archive
2010-04-27 15:09:21 CDT :LOG: could not open file
"pg_xlog/00000001000000A000000002" (log file 160, segment 2): No such file
or directory
2010-04-27 15:09:21 CDT :LOG: redo done at A0/1000068
2010-04-27 15:09:21 CDT :PANIC: could not open file
"pg_xlog/00000001000000A000000001" (log file 160, segment 1): No such file
or directory
2010-04-27 15:09:26 CDT :LOG: startup process (PID 22490) was terminated by
signal 6
2010-04-27 15:09:26 CDT :LOG: aborting startup due to startup process
failure
2010-04-27 15:09:26 CDT :LOG: logger shutting down

--------------------

At this point the server will now enter a restart loop and constantly
generate log files like this :

2010-04-27 15:09:26 CDT :LOG: database system was interrupted while in
recovery at log time 2010-04-27 15:05:08 CDT
2010-04-27 15:09:26 CDT :HINT: If this has occurred more than once some
data may be corrupted and you may need to choose an earlier recovery target.
2010-04-27 15:09:26 CDT :LOG: starting archive recovery
2010-04-27 15:09:26 CDT :LOG: restore_command = "/opt/data/restore.sh
/opt/wal/archwalremote/%f %p"
2010-04-27 15:09:26 CDT :LOG: could not open file
"pg_xlog/00000001000000A000000001" (log file 160, segment 1): No such file
or directory
2010-04-27 15:09:26 CDT :LOG: invalid primary checkpoint record
2010-04-27 15:09:26 CDT :LOG: could not open file
"pg_xlog/000000010000009F000000BA" (log file 159, segment 186): No such file
or directory
2010-04-27 15:09:26 CDT :LOG: invalid secondary checkpoint record
2010-04-27 15:09:26 CDT :PANIC: could not locate a valid checkpoint record
2010-04-27 15:09:30 CDT :LOG: startup process (PID 24191) was terminated by
signal 6
2010-04-27 15:09:30 CDT :LOG: aborting startup due to startup process
failure
2010-04-27 15:09:30 CDT :LOG: logger shutting down

--------------------

Any help is greatly appreciated. Please let me know if I can provide any
more information that will be helpful.

-Chris

Responses

Browse pgsql-general by date

  From Date Subject
Next Message John R Pierce 2010-04-27 20:40:27 Re: Open Source BI Tool
Previous Message Thom Brown 2010-04-27 20:38:26 Re: Errors starting postgres after initializing new database