Slave promotion failure

From: François Beausoleil <francois(at)teksol(dot)info>
To: Forums postgresql <pgsql-general(at)postgresql(dot)org>
Subject: Slave promotion failure
Date: 2013-06-06 16:37:29
Message-ID: 9F3B0CFA-D0AA-4783-BF55-618D85FE37C9@teksol.info
Views: Raw Message | Whole Thread | Download mbox | Resend email
Thread:
Lists: pgsql-general

Hi,

I have the following recovery.conf (Ubuntu 12.04):

standby_mode = on
restore_command = '/usr/local/omnipitr/bin/omnipitr-restore -D /var/lib/postgresql/9.1/main/ --source gzip=/var/backups/seevibes/wal/dbanalytics.production/ --remove-unneeded --temp-dir /var/tmp/omnipitr -l /var/log/omnipitr/restore.log --error-pgcontroldata hang --pgcontroldata-path /usr/lib/postgresql/9.1/bin/pg_controldata "%f" "%p"'
trigger_file = '/var/lib/postgresql/9.1/main/recovery.done'
archive_cleanup_command = '/usr/local/omnipitr/bin/omnipitr-cleanup --log /var/log/omnipitr/cleanup.log --archive gzip=/var/backups/seevibes/wal/dbanalytics.production "%r"'

I can't seem to promote the slave:

$ sudo -u postgres touch /var/lib/postgresql/9.1/main/recovery.done
# log is silent

$ sudo -u postgres /usr/lib/postgresql/9.1/bin/pg_ctl promote -D /var/lib/postgresql/9.1/main
server promoting
# log is silent

The postgresql around the time I attempted the promotions is:

2013-06-06 16:21:51.030 UTC - @ 26434 (00000) 2013-06-06 16:20:42 UTC - LOG: restored log file "0000000400001658000000CB" from archive
2013-06-06 16:22:35.324 UTC - @ 26411 (00000) 2013-06-06 16:20:41 UTC - LOG: received SIGHUP, reloading configuration files
2013-06-06 16:22:51.457 UTC - @ 26434 (00000) 2013-06-06 16:20:42 UTC - LOG: restored log file "0000000400001658000000CC" from archive
2013-06-06 16:24:51.034 UTC - @ 26434 (00000) 2013-06-06 16:20:42 UTC - LOG: restored log file "0000000400001658000000CD" from archive

The SIGHUP occurred because recovery.conf wasn't owned by user postgres (using Puppet and configuration reloads on any change). A quick scan of the data directory reveals nothing out of the ordinary:

$ sudo ls -la /var/lib/postgresql/9.1/main/
total 108
drwx------ 13 postgres postgres 4096 Jun 6 16:27 .
drwxr-xr-x 3 700 postgres 17 May 31 15:46 ..
-rw------- 1 postgres postgres 184 May 22 12:10 backup_label.old
drwx------ 12 postgres postgres 130 Apr 18 17:33 base
drwx------ 2 postgres postgres 8192 Jun 6 16:32 global
drwx------ 2 postgres postgres 4096 May 31 20:57 pg_clog
drwx------ 4 postgres postgres 34 Jan 12 2012 pg_multixact
drwx------ 2 postgres postgres 17 Jun 6 16:20 pg_notify
drwx------ 2 postgres postgres 6 Jan 12 2012 pg_serial
drwx------ 2 postgres postgres 24 Jun 6 16:20 pg_stat_tmp
drwx------ 2 postgres postgres 17 Jun 6 08:32 pg_subtrans
drwx------ 2 postgres postgres 6 Jan 12 2012 pg_tblspc
drwx------ 2 postgres postgres 6 Jan 12 2012 pg_twophase
-rw------- 1 postgres postgres 4 Jan 12 2012 PG_VERSION
drwxr-xr-x 3 postgres postgres 45056 Jun 6 16:34 pg_xlog
-rw------- 1 postgres postgres 350 Jun 6 16:20 postmaster.opts
-rw------- 1 postgres postgres 93 Jun 6 16:20 postmaster.pid
-rw-r--r-- 1 postgres postgres 591 Jun 6 16:20 recovery.conf
lrwxrwxrwx 1 postgres postgres 36 Dec 2 2012 server.crt -> /etc/ssl/certs/ssl-cert-snakeoil.pem
lrwxrwxrwx 1 postgres postgres 38 Dec 2 2012 server.key -> /etc/ssl/private/ssl-cert-snakeoil.key

I also attempted to restart the slave, with and without recovery.done, to no avail. I must be missing something. Someone has an idea? I did read http://www.postgresql.org/docs/9.1/static/warm-standby-failover.html very carefully. I believe I did everything I was supposed to do.

Thanks,
François Beausoleil

Responses

Browse pgsql-general by date

  From Date Subject
Next Message Jeff Janes 2013-06-06 16:51:58 Re: Trouble with replication
Previous Message Jeff Janes 2013-06-06 16:32:55 Re: Streaming replication with sync slave, but disconnects due to missing WAL segments