Does my pg_xlog directory look right?

Hey all,

My pg_xlog directory looks kind of messed up. Specifically:

1) There's more segment files than I would expect;
2) Segment files appear to be out of order with respect to filename and
timestamp

System info:
--------------
OS: Ubuntu 14.04 Linux 3.13.0-83-generic
PG Version: 9.3.11
- I have streaming replication set up to one remote, and am archiving to
another remote.

postgresql.conf
--------------
wal_level = hot_standby
checkpoint_segments = 32
checkpoint_completion_target = 0.7
archive_mode = on
archive_command = 'rsync -a %p user(at)host:/dir/to/barman/main/incoming/%f'
wal_keep_segments = 32

Here is an `ls -l` of the xlog directory:

-rw------- 1 postgres postgres 43 Oct 10 2014 00000002.history
-rw------- 1 postgres postgres 43 Oct 10 2014 00000003.history
-rw------- 1 postgres postgres 43 Mar 9 2015 00000004.history
-rw------- 1 postgres postgres 87 Mar 10 2015 00000005.history
-rw------- 1 postgres postgres 131 Aug 16 2015 00000006.history
-rw------- 1 postgres postgres 175 Aug 17 2015 00000007.history
-rw------- 1 postgres postgres 220 Dec 20 09:17 00000008.history
-rw------- 1 postgres postgres 265 Dec 21 01:17 00000009.history
-rw------- 1 postgres postgres 310 Jan 19 01:49 0000000A.history
-rw------- 1 postgres postgres 356 Jan 19 02:37 0000000B.history
-rw------- 1 postgres postgres 402 Feb 22 01:59 0000000C.history
-rw------- 1 postgres postgres 448 Feb 22 02:51 0000000D.history
-rw------- 1 postgres postgres 494 Mar 27 04:10 0000000E.history
-rw------- 1 postgres postgres 540 Mar 27 08:20 0000000F.history
-rw------- 1 postgres postgres 586 Mar 27 22:32 00000010.history
-rw------- 1 postgres postgres 632 Apr 3 23:03 00000011.history
-rw------- 1 postgres postgres 327 May 20 02:21
000000120000037000000089.00000028.backup
-rw------- 1 postgres postgres 16777216 May 20 15:12
00000012000003720000002D
-rw------- 1 postgres postgres 16777216 May 20 15:15
00000012000003720000002E
-rw------- 1 postgres postgres 16777216 May 20 15:18
00000012000003720000002F
-rw------- 1 postgres postgres 16777216 May 20 15:21
000000120000037200000030
-rw------- 1 postgres postgres 16777216 May 20 15:22
000000120000037200000031
-rw------- 1 postgres postgres 16777216 May 20 15:24
000000120000037200000032
-rw------- 1 postgres postgres 16777216 May 20 15:26
000000120000037200000033
-rw------- 1 postgres postgres 16777216 May 20 15:28
000000120000037200000034
-rw------- 1 postgres postgres 16777216 May 20 15:32
000000120000037200000035
-rw------- 1 postgres postgres 16777216 May 20 15:33
000000120000037200000036
-rw------- 1 postgres postgres 16777216 May 20 15:35
000000120000037200000037
-rw------- 1 postgres postgres 16777216 May 20 15:37
000000120000037200000038
-rw------- 1 postgres postgres 16777216 May 20 15:38
000000120000037200000039
-rw------- 1 postgres postgres 16777216 May 20 15:39
00000012000003720000003A
-rw------- 1 postgres postgres 16777216 May 20 15:39
00000012000003720000003B
-rw------- 1 postgres postgres 16777216 May 20 15:39
00000012000003720000003C
-rw------- 1 postgres postgres 16777216 May 20 15:43
00000012000003720000003D
-rw------- 1 postgres postgres 16777216 May 20 15:45
00000012000003720000003E
-rw------- 1 postgres postgres 16777216 May 20 15:48
00000012000003720000003F
-rw------- 1 postgres postgres 16777216 May 20 15:50
000000120000037200000040
-rw------- 1 postgres postgres 16777216 May 20 15:52
000000120000037200000041
-rw------- 1 postgres postgres 16777216 May 20 15:52
000000120000037200000042
-rw------- 1 postgres postgres 16777216 May 20 15:53
000000120000037200000043
-rw------- 1 postgres postgres 16777216 May 20 15:54
000000120000037200000044
-rw------- 1 postgres postgres 16777216 May 20 15:55
000000120000037200000045
-rw------- 1 postgres postgres 16777216 May 20 15:56
000000120000037200000046
-rw------- 1 postgres postgres 16777216 May 20 15:58
000000120000037200000047
-rw------- 1 postgres postgres 16777216 May 20 16:01
000000120000037200000048
-rw------- 1 postgres postgres 16777216 May 20 16:02
000000120000037200000049
-rw------- 1 postgres postgres 16777216 May 20 16:02
00000012000003720000004A
-rw------- 1 postgres postgres 16777216 May 20 16:03
00000012000003720000004B
-rw------- 1 postgres postgres 16777216 May 20 16:03
00000012000003720000004C
-rw------- 1 postgres postgres 16777216 May 20 16:04
00000012000003720000004D
-rw------- 1 postgres postgres 16777216 May 20 16:04
00000012000003720000004E
-rw------- 1 postgres postgres 16777216 May 20 16:04
00000012000003720000004F
-rw------- 1 postgres postgres 16777216 May 20 16:04
000000120000037200000050
-rw------- 1 postgres postgres 16777216 May 20 16:04
000000120000037200000051
-rw------- 1 postgres postgres 16777216 May 20 16:04
000000120000037200000052
-rw------- 1 postgres postgres 16777216 May 20 16:04
000000120000037200000053
-rw------- 1 postgres postgres 16777216 May 20 16:04
000000120000037200000054
-rw------- 1 postgres postgres 16777216 May 20 16:04
000000120000037200000055
-rw------- 1 postgres postgres 16777216 May 20 16:04
000000120000037200000056
-rw------- 1 postgres postgres 16777216 May 20 16:05
000000120000037200000057
-rw------- 1 postgres postgres 16777216 May 20 16:06
000000120000037200000058
-rw------- 1 postgres postgres 16777216 May 20 16:06
000000120000037200000059
-rw------- 1 postgres postgres 16777216 May 20 16:06
00000012000003720000005A
-rw------- 1 postgres postgres 16777216 May 20 16:07
00000012000003720000005B
-rw------- 1 postgres postgres 16777216 May 20 16:07
00000012000003720000005C
-rw------- 1 postgres postgres 16777216 May 20 16:08
00000012000003720000005D
-rw------- 1 postgres postgres 16777216 May 20 14:12
00000012000003720000005E
-rw------- 1 postgres postgres 16777216 May 20 14:12
00000012000003720000005F
-rw------- 1 postgres postgres 16777216 May 20 14:12
000000120000037200000060
-rw------- 1 postgres postgres 16777216 May 20 14:12
000000120000037200000061
-rw------- 1 postgres postgres 16777216 May 20 14:13
000000120000037200000062
-rw------- 1 postgres postgres 16777216 May 20 14:15
000000120000037200000063
-rw------- 1 postgres postgres 16777216 May 20 14:12
000000120000037200000064
-rw------- 1 postgres postgres 16777216 May 20 14:12
000000120000037200000065
-rw------- 1 postgres postgres 16777216 May 20 14:12
000000120000037200000066
-rw------- 1 postgres postgres 16777216 May 20 14:19
000000120000037200000067
-rw------- 1 postgres postgres 16777216 May 20 14:17
000000120000037200000068
-rw------- 1 postgres postgres 16777216 May 20 14:19
000000120000037200000069
-rw------- 1 postgres postgres 16777216 May 20 14:19
00000012000003720000006A
-rw------- 1 postgres postgres 16777216 May 20 14:20
00000012000003720000006B
-rw------- 1 postgres postgres 16777216 May 20 14:20
00000012000003720000006C
-rw------- 1 postgres postgres 16777216 May 20 14:20
00000012000003720000006D
-rw------- 1 postgres postgres 16777216 May 20 14:21
00000012000003720000006E
-rw------- 1 postgres postgres 16777216 May 20 14:21
00000012000003720000006F
-rw------- 1 postgres postgres 16777216 May 20 14:21
000000120000037200000070
-rw------- 1 postgres postgres 16777216 May 20 14:21
000000120000037200000071
-rw------- 1 postgres postgres 16777216 May 20 14:22
000000120000037200000072
-rw------- 1 postgres postgres 16777216 May 20 14:24
000000120000037200000073
-rw------- 1 postgres postgres 16777216 May 20 14:27
000000120000037200000074
-rw------- 1 postgres postgres 16777216 May 20 14:25
000000120000037200000075
-rw------- 1 postgres postgres 16777216 May 20 14:25
000000120000037200000076
-rw------- 1 postgres postgres 16777216 May 20 14:30
000000120000037200000077
-rw------- 1 postgres postgres 16777216 May 20 14:32
000000120000037200000078
-rw------- 1 postgres postgres 16777216 May 20 14:34
000000120000037200000079
-rw------- 1 postgres postgres 16777216 May 20 14:36
00000012000003720000007A
-rw------- 1 postgres postgres 16777216 May 20 14:37
00000012000003720000007B
-rw------- 1 postgres postgres 16777216 May 20 14:40
00000012000003720000007C
-rw------- 1 postgres postgres 16777216 May 20 14:41
00000012000003720000007D
-rw------- 1 postgres postgres 16777216 May 20 14:43
00000012000003720000007E
-rw------- 1 postgres postgres 16777216 May 20 14:43
00000012000003720000007F
-rw------- 1 postgres postgres 16777216 May 20 14:43
000000120000037200000080
-rw------- 1 postgres postgres 16777216 May 20 14:44
000000120000037200000081
-rw------- 1 postgres postgres 16777216 May 20 14:45
000000120000037200000082
-rw------- 1 postgres postgres 16777216 May 20 14:45
000000120000037200000083
-rw------- 1 postgres postgres 16777216 May 20 14:45
000000120000037200000084
-rw------- 1 postgres postgres 16777216 May 20 14:45
000000120000037200000085
-rw------- 1 postgres postgres 16777216 May 20 14:50
000000120000037200000086
-rw------- 1 postgres postgres 16777216 May 20 14:51
000000120000037200000087
-rw------- 1 postgres postgres 16777216 May 20 14:55
000000120000037200000088
-rw------- 1 postgres postgres 16777216 May 20 14:49
000000120000037200000089
-rw------- 1 postgres postgres 16777216 May 20 15:06
00000012000003720000008A
-rw------- 1 postgres postgres 16777216 May 20 14:59
00000012000003720000008B
-rw------- 1 postgres postgres 16777216 May 20 15:10
00000012000003720000008C
-rw------- 1 postgres postgres 16777216 May 20 15:02
00000012000003720000008D
-rw------- 1 postgres postgres 16777216 May 20 15:08
00000012000003720000008E
-rw------- 1 postgres postgres 678 Apr 3 23:56 00000012.history
drwx------ 2 postgres postgres 4096 May 20 16:07 archive_status

There are 98 WAL segments in the directory currently.

From the documentation, I understand that the number of WAL files can go as
high as (3 * checkpoint_segments + 1) segment files, which would be 97 in
my case, but I seem to be consistently at this number or even higher
(sometimes over 110 files) and I don't think I've ever seen the directory
smaller than ~97 segment files.

Regarding the ordering of segments, my understanding is that segment
0000012000003720000008B would be written to after 00000012000003720000008A,
but the timestamps (14:59 and 15:06, respectively) appear to show
otherwise. There are several cases like this. Does this mean that
00000012000003720000008A has newer WAL records than
00000012000003720000008B??

It may be that there's nothing wrong here and this behaviour is natural. If
that is the case, I'd still like to better understand this phenomena.

Thanks!

Dave Stibrany