WAL replay asking for very old WAL

From: Stephen Kuntz <Skuntz(at)pelmorex(dot)com>
To: "pgsql-admin(at)postgresql(dot)org" <pgsql-admin(at)postgresql(dot)org>
Subject: WAL replay asking for very old WAL
Date: 2016-03-07 15:37:51
Message-ID: DM2PR0401MB10884C9A50BD2B474ABB2E5BACB10@DM2PR0401MB1088.namprd04.prod.outlook.com
Views: Raw Message | Whole Thread | Download mbox | Resend email
Thread:
Lists: pgsql-admin

Hello,

I have a 2 servers that are geographically separated be about 570KM which are connected via LAN extension. The slave at location 2 fell behind and I've had to restore it with pg_basebackup then replay my WAL files. Part way through the replay I get an error and then it starts asking for an old out of sequence WAL file that I no longer have. It continues to restore the WAL files in the sequence but continually askes for the old one, even after it has caught up to the live environment. I've tried it three separate times and it is always the same file name. There is un unlisten command that seems to trigger this to happen, but I'm not sure where it came from.

Any help is appreciated.

PostgreSQL 9.3.3

...
2016-03-06 00:25:45 UTC LOG: restored log file "000000010000578800000038" from archive
2016-03-06 00:25:45 UTC FATAL: the database system is starting up
2016-03-06 00:25:45 UTC FATAL: the database system is starting up
2016-03-06 00:25:45 UTC FATAL: the database system is starting up
2016-03-06 00:25:45 UTC FATAL: the database system is starting up
2016-03-06 00:25:45 UTC FATAL: the database system is starting up
2016-03-06 00:25:47 UTC LOG: restored log file "000000010000578800000039" from archive
2016-03-06 00:25:48 UTC LOG: restored log file "00000001000057880000003A" from archive
2016-03-06 00:25:49 UTC FATAL: the database system is starting up
2016-03-06 00:25:50 UTC LOG: restored log file "00000001000057880000003B" from archive
2016-03-06 00:25:50 UTC FATAL: the database system is starting up
2016-03-06 00:25:51 UTC LOG: restored log file "00000001000057880000003C" from archive
2016-03-06 00:25:53 UTC LOG: restored log file "00000001000057880000003D" from archive
2016-03-06 00:25:53 UTC LOG: consistent recovery state reached at 5788/3DCB5EF8
2016-03-06 00:25:53 UTC LOG: database system is ready to accept read only connections
2016-03-06 00:25:54 UTC ERROR: cannot execute UNLISTEN during recovery
2016-03-06 00:25:54 UTC STATEMENT: unlisten *
2016-03-06 00:25:54 UTC ERROR: cannot execute UNLISTEN during recovery
2016-03-06 00:25:54 UTC STATEMENT: unlisten *
2016-03-06 00:25:54 UTC LOG: restored log file "00000001000057880000003E" from archive
2016-03-06 00:25:54 UTC ERROR: requested WAL segment 00000001000054540000007E has already been removed
2016-03-06 00:25:55 UTC ERROR: cannot execute UNLISTEN during recovery
2016-03-06 00:25:55 UTC STATEMENT: unlisten *
2016-03-06 00:25:55 UTC ERROR: cannot execute UNLISTEN during recovery
2016-03-06 00:25:55 UTC STATEMENT: unlisten *
2016-03-06 00:25:55 UTC ERROR: cannot execute UNLISTEN during recovery
2016-03-06 00:25:55 UTC STATEMENT: unlisten *
2016-03-06 00:25:55 UTC ERROR: cannot execute UNLISTEN during recovery
2016-03-06 00:25:55 UTC STATEMENT: unlisten *
2016-03-06 00:25:56 UTC LOG: restored log file "00000001000057880000003F" from archive
2016-03-06 00:25:57 UTC LOG: restored log file "000000010000578800000040" from archive
2016-03-06 00:25:58 UTC LOG: restored log file "000000010000578800000041" from archive
2016-03-06 00:26:00 UTC ERROR: requested WAL segment 00000001000054540000007E has already been removed
2016-03-06 00:26:00 UTC LOG: restored log file "000000010000578800000042" from archive
2016-03-06 00:26:00 UTC ERROR: cannot execute UNLISTEN during recovery
2016-03-06 00:26:00 UTC STATEMENT: unlisten *
2016-03-06 00:26:01 UTC LOG: restored log file "000000010000578800000043" from archive
2016-03-06 00:26:02 UTC LOG: restored log file "000000010000578800000044" from archive
2016-03-06 00:26:03 UTC ERROR: cannot execute UNLISTEN during recovery
2016-03-06 00:26:03 UTC STATEMENT: unlisten *
2016-03-06 00:26:03 UTC ERROR: cannot execute UNLISTEN during recovery
2016-03-06 00:26:03 UTC STATEMENT: unlisten *
2016-03-06 00:26:03 UTC ERROR: cannot execute UNLISTEN during recovery
2016-03-06 00:26:03 UTC STATEMENT: unlisten *
2016-03-06 00:26:03 UTC LOG: restored log file "000000010000578800000045" from archive
2016-03-06 00:26:04 UTC LOG: restored log file "000000010000578800000046" from archive
2016-03-06 00:26:05 UTC ERROR: requested WAL segment 00000001000054540000007E has already been removed
2016-03-06 00:26:06 UTC LOG: restored log file "000000010000578800000047" from archive
2016-03-06 00:26:07 UTC LOG: restored log file "000000010000578800000048" from archive
2016-03-06 00:26:07 UTC LOG: restored log file "000000010000578800000049" from archive
2016-03-06 00:26:09 UTC LOG: restored log file "00000001000057880000004A" from archive
2016-03-06 00:26:10 UTC ERROR: requested WAL segment 00000001000054540000007E has already been removed....

Responses

Browse pgsql-admin by date

  From Date Subject
Next Message MirrorX 2016-03-07 15:48:23 Re: WAL replay asking for very old WAL
Previous Message Nagy László Zsolt 2016-03-06 09:53:04 Re: How to setup a good collation?