pg_rewind - restore new slave failed to startup during recovery

Hi
I have 1 master and 1 slave wal streaming replication setup and the Application connects via a load balancer (LTM) where the all connections are redirected to the master member (master db).

We have archive_mode enabled.

I am trying to test to use pg_rewind to restore the new slave (old master) after a failover while the system is under load.

Here are the steps I take to test:

1. Disable the master ltm member (all connections redired to slave member)

2. Promote slave (touch promote.me)

3. Stop the master db (old master)

4. Do pg_rewind on the new slave (old master)

5. Start the new slave.

Here are my results:
[cid:image001(dot)png(at)01D31AA0(dot)A97716C0]

However, when I tried to start the new slave, I am getting the error that it cannot locate the archive wal files and can not receive data from WAL stream error:
[cid:image002(dot)png(at)01D31AA1(dot)9D1DA960]

Checking the on the new master, I see that the check point that its trying to restore is the file 000000040000009C0000006F, but the file does not exist anywhere on the new master. Not in the pg_xlog or the archive folder. (as specified in the postgresql.conf)

[cid:image003(dot)png(at)01D31AA1(dot)9D1DA960]

Here is my recovery.conf :

standby_mode = 'on'
primary_conninfo = 'host=10.69.19.18 user=replicant'
trigger_file = '/var/run/promote_me'
restore_command = 'cp /pg_backup/backup/archive_sync/%f "%p"'

does anyone know why?

Under what conditions will pg_rewind wont' work?

Thanks
Dylan