Re: [GENERAL] pg_xlog on a hot_standby slave filling up

Re: Jeff Frost 2015-06-16 <67E2F20A-6A2E-484E-BF97-544F1FC66566(at)pgexperts(dot)com>
> I’ve seen this before, but haven’t been able to make a reproducible test case yet.
>
> Are you by chance using SSL to talk to the primary server? Is the ssl_renegotiation_limit the default of 512MB? 32 WAL files at 16MB each = 512MB. I found that it would always leave the WAL file from before the invalid record length message. Does that seem to be the case for you as well?

Yes, SSL, with default settings. I can confirm your
wal-file-from-before analysis:

Jun 16 07:14:59 synthesis postgres[32525]: [8-1] 2015-06-16 07:14:59 GMT LOG: unexpected pageaddr 33/39000000 in log segment 000000010000003300000040, offset 0
Jun 16 07:15:00 synthesis postgres[11514]: [3-1] 2015-06-16 07:15:00 GMT LOG: started streaming WAL from primary at 33/40000000 on timeline 1

-rw------- 1 postgres postgres 16777216 Jun 16 09:14 00000001000000330000003F

Jun 16 17:55:01 synthesis postgres[32525]: [9-1] 2015-06-16 17:55:01 GMT LOG: unexpected pageaddr 33/5A000000 in log segment 000000010000003300000060, offset 0
Jun 16 17:55:02 synthesis postgres[24337]: [3-1] 2015-06-16 17:55:02 GMT LOG: started streaming WAL from primary at 33/60000000 on timeline 1

-rw------- 1 postgres postgres 16777216 Jun 16 19:55 00000001000000330000005F

Jun 17 04:35:02 synthesis postgres[24337]: [4-1] 2015-06-17 04:35:02 GMT FATAL: could not send data to WAL stream: server closed the connection unexpectedly
Jun 17 04:35:02 synthesis postgres[24337]: [4-2] This probably means the server terminated abnormally
Jun 17 04:35:02 synthesis postgres[24337]: [4-3] before or while processing the request.
Jun 17 04:35:02 synthesis postgres[24337]: [4-4]
Jun 17 04:35:04 synthesis postgres[32525]: [10-1] 2015-06-17 04:35:04 GMT LOG: unexpected pageaddr 33/7B000000 in log segment 000000010000003300000080, offset 0
Jun 17 04:35:05 synthesis postgres[4756]: [5-1] 2015-06-17 04:35:05 GMT LOG: started streaming WAL from primary at 33/80000000 on timeline 1

-rw------- 1 postgres postgres 16777216 Jun 17 06:35 00000001000000330000007F

There's a 1:1 correspondence with log and leaked files.

Christoph
--
cb(at)df7cb(dot)de | http://www.df7cb.de/