Trying to reduce xlog wal files

I am running Postgresql 9.3.15 and I am trying to reduce the amount of
wal files being recycled in pg_xlog. Archive is set up and working
correctly. A while ago we had problems with the streaming slave falling
behind because of hardware being slower and tuned postgres to keep over
1000k wal files so it could catch up without us manually moving files
from the backup server. We have since upgraded the slave so it never
falls behind and would like to reduce the wals files kept in pg_xlog. I
have updated postgresql.conf checkpoint_segments from 1000 to 250 and
checkpoint_timeout from 16min to 5min. I ran SELECT pg_reload_conf();
and using pg_controldata I have confirmed the settings and checkpoints
are happening at 5 min intervals. However, file count in pg_xlog stays
at 1152 and never changes. How can I get this to reduce the file count?

Thanks