From: | Mathieu Fenniak <mathieu(dot)fenniak(at)replicon(dot)com> |
---|---|
To: | pgsql-hackers <pgsql-hackers(at)postgresql(dot)org> |
Subject: | restartpoints stop generating on streaming-replication slave |
Date: | 2012-08-21 21:21:56 |
Message-ID: | CAHoiPjxe6DNO9mr6TKb4p-jcRibjzRnPOXoj1M4a-2bSn266PQ@mail.gmail.com |
Views: | Raw Message | Whole Thread | Download mbox | Resend email |
Thread: | |
Lists: | pgsql-hackers |
Hi all,
I've been investigating an issue with our PostgreSQL 9.1.1 (Linux x86-64
CentOS 5.8) database where restartpoints suddenly stop being generated on
the streaming-replication slave after working correctly for a week or two.
The symptom of the problem is that the pg_xlog directory on the slave
doesn't get cleaned up, and the log_checkpoints output (eg. restartpoint
starting: time) stops appearing.
I was able to extract a core dump of the bgwriter process while it was in
BgWriterNap. I inspected ckpt_start_time and last_checkpoint_time;
ckpt_start_time was 1345578533 (... 19:48:53 GMT) and last_checkpoint_time
was 1345578248 (... 19:44:08 GMT). Based upon these values, I concluded
that it's performing checkpoints but missing the "if (ckpt_performed)"
condition (ie. CreateRestartPoint returns false); it's then setting
last_checkpoint_time to now - 5 minutes + 15 seconds.
There seems to be two causes of a false retval in CreateRestartPoint; the
first is if !RecoveryInProgress(), and the second is if "the last
checkpoint record we've replayed is already our last restartpoint". The
first condition doesn't seem likely; does anyone know how we might be
hitting the second condition? We have continuous traffic on the master
server in the range of 1000 txn/s, and the slave seems to be completely
up-to-date, so I don't understand how we could be hitting this condition.
Mathieu
From | Date | Subject | |
---|---|---|---|
Next Message | Tatsuo Ishii | 2012-08-21 21:26:00 | Re: multi-master pgbench? |
Previous Message | Robert Haas | 2012-08-21 21:20:35 | Re: reviewing the "Reduce sinval synchronization overhead" patch / b4fbe392f8ff6ff1a66b488eb7197eef9e1770a4 |