BUG #13362: pg_basebackup not working in 9.4.2

The following bug has been logged on the website:

Bug reference: 13362
Logged by: CJ Estel
Email address: cestel(at)covermymeds(dot)com
PostgreSQL version: 9.4.2
Operating system: RHEL7
Description:

#SUMMARY
On postgres 9.4.2 I can successfully run pg_basebackup, but then am unable
to start the postgres service with permission related issues. I first get
an error on a file that is there due to our log rotation process. This file
is owned by postgres and is readable. If I remove that file (as it isn't
needed by postgres in any way), it then gives me a similar permission error
on my server.crt SSL file. This file is also owned by postgres and
readable. Performing the same steps on 9.4.1, I do not have any issues.
Below is output from my SOURCE and DESTINATION servers. In the examples
provided, I am not setting up a recovery.conf file, but in my tests with
that it does not improve the likelihood that I will be able to start the
server.

#SOURCE VERSION
root(at)pgtest2:TESTING:cestel> rpm -q postgresql94
postgresql94-9.4.1-1PGDG.rhel7.x86_64

#DESTINATION PG_BASEBACKUP WORKS CORRECTLY
root(at)g1-cestel:DEVELOPMENT:9.4> service postgresql-9.4 stop
Redirecting to /bin/systemctl stop postgresql-9.4.service
root(at)g1-cestel:DEVELOPMENT:9.4> rpm -q postgresql94
postgresql94-9.4.1-1PGDG.rhel7.x86_64
root(at)g1-cestel:DEVELOPMENT:9.4> pwd
/var/lib/pgsql/9.4
root(at)g1-cestel:DEVELOPMENT:9.4> rm -rf data/*
root(at)g1-cestel:DEVELOPMENT:9.4> su - postgres -c "pg_basebackup -D
/var/lib/pgsql/9.4/data --host=
pgtest2 --port=5432 -x -P --username=repl --password"
Password:
2639671/2639671 kB (100%), 1/1 tablespace
root(at)g1-cestel:DEVELOPMENT:9.4> service postgresql-9.4 restart
Redirecting to /bin/systemctl restart postgresql-9.4.service
root(at)g1-cestel:DEVELOPMENT:9.4> su postgres -c psql
psql (9.4.1)
Type "help" for help.

postgres=# \q

#UPGRADE DESTINATION TO 9.4.2
root(at)g1-cestel:DEVELOPMENT:9.4> yum upgrade postgresql94 postgresql94-devel
postgresql94-libs postgresql94-server postgresql94-contrib
Loaded plugins: langpacks, product-id, rhnplugin, subscription-manager
This system is receiving updates from RHN Classic or Red Hat Satellite.
Resolving Dependencies
--> Running transaction check
---> Package postgresql94.x86_64 0:9.4.1-1PGDG.rhel7 will be updated
---> Package postgresql94.x86_64 0:9.4.2-2PGDG.rhel7 will be an update
---> Package postgresql94-contrib.x86_64 0:9.4.1-1PGDG.rhel7 will be
updated
---> Package postgresql94-contrib.x86_64 0:9.4.2-2PGDG.rhel7 will be an
update
---> Package postgresql94-devel.x86_64 0:9.4.1-1PGDG.rhel7 will be updated
---> Package postgresql94-devel.x86_64 0:9.4.2-2PGDG.rhel7 will be an
update
---> Package postgresql94-libs.x86_64 0:9.4.1-1PGDG.rhel7 will be updated
---> Package postgresql94-libs.x86_64 0:9.4.2-2PGDG.rhel7 will be an update
---> Package postgresql94-server.x86_64 0:9.4.1-1PGDG.rhel7 will be updated
---> Package postgresql94-server.x86_64 0:9.4.2-2PGDG.rhel7 will be an
update
--> Finished Dependency Resolution

Dependencies Resolved

===================================================================================================
Package Arch Version
Repository Size
===================================================================================================
Updating:
postgresql94 x86_64 9.4.2-2PGDG.rhel7
yum.postgresql.org 1.0 M
postgresql94-contrib x86_64 9.4.2-2PGDG.rhel7
yum.postgresql.org 609 k
postgresql94-devel x86_64 9.4.2-2PGDG.rhel7
yum.postgresql.org 1.6 M
postgresql94-libs x86_64 9.4.2-2PGDG.rhel7
yum.postgresql.org 208 k
postgresql94-server x86_64 9.4.2-2PGDG.rhel7
yum.postgresql.org 3.9 M

Transaction Summary
===================================================================================================
Upgrade 5 Packages

Total download size: 7.3 M
Is this ok [y/d/N]: y
Downloading packages:
No Presto metadata available for yum.postgresql.org
(1/5): postgresql94-9.4.2-2PGDG.rhel7.x86_64.rpm
| 1.0 MB 00:00:00
(2/5): postgresql94-contrib-9.4.2-2PGDG.rhel7.x86_64.rpm
| 609 kB 00:00:00
(3/5): postgresql94-libs-9.4.2-2PGDG.rhel7.x86_64.rpm
| 208 kB 00:00:00
(4/5): postgresql94-devel-9.4.2-2PGDG.rhel7.x86_64.rpm
| 1.6 MB 00:00:00
(5/5): postgresql94-server-9.4.2-2PGDG.rhel7.x86_64.rpm
| 3.9 MB 00:00:01
---------------------------------------------------------------------------------------------------
Total 2.6 MB/s
| 7.3 MB 00:00:02
Running transaction check
Running transaction test
Transaction test succeeded
Running transaction
Updating : postgresql94-libs-9.4.2-2PGDG.rhel7.x86_64
1/10
Updating : postgresql94-9.4.2-2PGDG.rhel7.x86_64
2/10
Updating : postgresql94-devel-9.4.2-2PGDG.rhel7.x86_64
3/10
Updating : postgresql94-server-9.4.2-2PGDG.rhel7.x86_64
4/10
Updating : postgresql94-contrib-9.4.2-2PGDG.rhel7.x86_64
5/10
Cleanup : postgresql94-contrib-9.4.1-1PGDG.rhel7.x86_64
6/10
Cleanup : postgresql94-server-9.4.1-1PGDG.rhel7.x86_64
7/10
Cleanup : postgresql94-devel-9.4.1-1PGDG.rhel7.x86_64
8/10
Cleanup : postgresql94-9.4.1-1PGDG.rhel7.x86_64
9/10
Cleanup : postgresql94-libs-9.4.1-1PGDG.rhel7.x86_64
10/10
Verifying : postgresql94-devel-9.4.2-2PGDG.rhel7.x86_64
1/10
Verifying : postgresql94-9.4.2-2PGDG.rhel7.x86_64
2/10
Verifying : postgresql94-server-9.4.2-2PGDG.rhel7.x86_64
3/10
Verifying : postgresql94-contrib-9.4.2-2PGDG.rhel7.x86_64
4/10
Verifying : postgresql94-libs-9.4.2-2PGDG.rhel7.x86_64
5/10
Verifying : postgresql94-libs-9.4.1-1PGDG.rhel7.x86_64
6/10
Verifying : postgresql94-devel-9.4.1-1PGDG.rhel7.x86_64
7/10
Verifying : postgresql94-server-9.4.1-1PGDG.rhel7.x86_64
8/10
Verifying : postgresql94-9.4.1-1PGDG.rhel7.x86_64
9/10
Verifying : postgresql94-contrib-9.4.1-1PGDG.rhel7.x86_64
10/10

Updated:
postgresql94.x86_64 0:9.4.2-2PGDG.rhel7
postgresql94-contrib.x86_64 0:9.4.2-2PGDG.rhel7
postgresql94-devel.x86_64 0:9.4.2-2PGDG.rhel7 postgresql94-libs.x86_64
0:9.4.2-2PGDG.rhel7
postgresql94-server.x86_64 0:9.4.2-2PGDG.rhel7

Complete!
root(at)g1-cestel:DEVELOPMENT:9.4> rpm -q postgresql94
postgresql94-9.4.2-2PGDG.rhel7.x86_64
root(at)g1-cestel:DEVELOPMENT:9.4> su postgres -c psql
psql (9.4.2)
Type "help" for help.

postgres=# \q

#TRY TO CREATE PG_BASEBACKUP WHILE DESTINATION IS 9.4.2 and SOURCE is 9.4.1
root(at)g1-cestel:DEVELOPMENT:9.4> service postgresql-9.4 stop
Redirecting to /bin/systemctl stop postgresql-9.4.service
root(at)g1-cestel:DEVELOPMENT:9.4> rm -rf data/*
root(at)g1-cestel:DEVELOPMENT:9.4> su - postgres -c "pg_basebackup -D
/var/lib/pgsql/9.4/data --host=pgtest2 --port=5432 -x -P --username=repl
--password"
Password:
2639776/2639776 kB (100%), 1/1 tablespace
root(at)g1-cestel:DEVELOPMENT:9.4> service postgresql-9.4 restart
Redirecting to /bin/systemctl restart postgresql-9.4.service
Job for postgresql-9.4.service failed. See 'systemctl status
postgresql-9.4.service' and 'journalctl -xn' for details.
root(at)g1-cestel:DEVELOPMENT:9.4> journalctl -xn
-- Logs begin at Thu 2015-05-21 14:13:26 EDT, end at Wed 2015-05-27 12:45:47
EDT. --
May 27 12:43:31 g1-cestel.dev.cmmint.net su[7765]: pam_unix(su-l:session):
session opened for user
May 27 12:44:29 g1-cestel.dev.cmmint.net su[7765]: pam_unix(su-l:session):
session closed for user
May 27 12:45:42 g1-cestel.dev.cmmint.net systemd[1]: Starting PostgreSQL 9.4
database server...
-- Subject: Unit postgresql-9.4.service has begun with start-up
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
--
-- Unit postgresql-9.4.service has begun starting up.
May 27 12:45:42 g1-cestel.dev.cmmint.net pg_ctl[7852]: 2015-05-27 12:45:42
EDT [7857]: [1-1] user=,
May 27 12:45:42 g1-cestel.dev.cmmint.net pg_ctl[7852]: 2015-05-27 12:45:42
EDT [7857]: [2-1] user=,
May 27 12:45:47 g1-cestel.dev.cmmint.net pg_ctl[7852]: pg_ctl: could not
start server
May 27 12:45:47 g1-cestel.dev.cmmint.net pg_ctl[7852]: Examine the log
output.
May 27 12:45:47 g1-cestel.dev.cmmint.net systemd[1]: postgresql-9.4.service:
control process exited
May 27 12:45:47 g1-cestel.dev.cmmint.net systemd[1]: Failed to start
PostgreSQL 9.4 database server
-- Subject: Unit postgresql-9.4.service has failed
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
--
-- Unit postgresql-9.4.service has failed.
--
-- The result is failed.
May 27 12:45:47 g1-cestel.dev.cmmint.net systemd[1]: Unit
postgresql-9.4.service entered failed sta
root(at)g1-cestel:DEVELOPMENT:9.4> cd data/pg_log
root(at)g1-cestel:DEVELOPMENT:pg_log> cat 2015-05-27_1245-postgresql.log
2015-05-27 12:45:42 EDT [7859]: [1-1] user=,db= LOG: database system was
interrupted; last known up at 2015-05-27 12:56:51 EDT
2015-05-27 12:45:43 EDT [7859]: [2-1] user=,db= FATAL: could not open file
"/var/lib/pgsql/9.4/data/pg_log/logrotate": Permission denied
2015-05-27 12:45:43 EDT [7857]: [3-1] user=,db= LOG: startup process (PID
7859) exited with exit code 1
2015-05-27 12:45:43 EDT [7857]: [4-1] user=,db= LOG: aborting startup due
to startup process failure
root(at)g1-cestel:DEVELOPMENT:pg_log> ls -lsa logrotate
0 -r--r--r-- 1 postgres postgres 0 May 27 12:44 logrotate
root(at)g1-cestel:DEVELOPMENT:pg_log> ls -lsa logrotate
0 -r--r--r-- 1 postgres postgres 0 May 27 12:44 logrotate
root(at)g1-cestel:DEVELOPMENT:pg_log> rm logrotate
rm: remove regular empty file ‘logrotate’? y
root(at)g1-cestel:DEVELOPMENT:pg_log> service postgresql-9.4 restart
Redirecting to /bin/systemctl restart postgresql-9.4.service
Job for postgresql-9.4.service failed. See 'systemctl status
postgresql-9.4.service' and 'journalctl -xn' for details.
root(at)g1-cestel:DEVELOPMENT:pg_log> cat 2015-05-27_1251-postgresql.log
2015-05-27 12:51:36 EDT [7902]: [1-1] user=,db= LOG: database system was
interrupted; last known up at 2015-05-27 12:56:51 EDT
2015-05-27 12:51:37 EDT [7902]: [2-1] user=,db= FATAL: could not open file
"/var/lib/pgsql/9.4/data/server.crt": Permission denied
2015-05-27 12:51:37 EDT [7900]: [3-1] user=,db= LOG: startup process (PID
7902) exited with exit code 1
2015-05-27 12:51:37 EDT [7900]: [4-1] user=,db= LOG: aborting startup due
to startup process failure
root(at)g1-cestel:DEVELOPMENT:pg_log> ls -lsa
/var/lib/pgsql/9.4/data/server.crt
4 -r-------- 1 postgres postgres 2297 May 27 12:44
/var/lib/pgsql/9.4/data/server.crt

#UPGRADE SOURCE
root(at)pgtest2:TESTING:cestel> yum update postgresql94
Loaded plugins: langpacks, product-id, rhnplugin, subscription-manager
This system is not registered to Red Hat Subscription Management. You can
use subscription-manager to register.
This system is receiving updates from RHN Classic or Red Hat Satellite.
cmm-rhel7
| 1.0 kB 00:00:00
rhel-x86_64-server-7
| 1.5 kB 00:00:00
rhel-x86_64-server-extras-7
| 1.3 kB 00:00:00
Resolving Dependencies
--> Running transaction check
---> Package postgresql94.x86_64 0:9.4.1-1PGDG.rhel7 will be updated
--> Processing Dependency: postgresql94 = 9.4.1-1PGDG.rhel7 for package:
postgresql94-server-9.4.1-1PGDG.rhel7.x86_64
--> Processing Dependency: postgresql94(x86-64) = 9.4.1-1PGDG.rhel7 for
package: postgresql94-devel-9.4.1-1PGDG.rhel7.x86_64
--> Processing Dependency: postgresql94(x86-64) = 9.4.1-1PGDG.rhel7 for
package: postgresql94-server-9.4.1-1PGDG.rhel7.x86_64
--> Processing Dependency: postgresql94(x86-64) = 9.4.1-1PGDG.rhel7 for
package: postgresql94-contrib-9.4.1-1PGDG.rhel7.x86_64
---> Package postgresql94.x86_64 0:9.4.2-2PGDG.rhel7 will be an update
--> Processing Dependency: postgresql94-libs = 9.4.2-2PGDG.rhel7 for
package: postgresql94-9.4.2-2PGDG.rhel7.x86_64
--> Running transaction check
---> Package postgresql94-contrib.x86_64 0:9.4.1-1PGDG.rhel7 will be
updated
---> Package postgresql94-contrib.x86_64 0:9.4.2-2PGDG.rhel7 will be an
update
---> Package postgresql94-devel.x86_64 0:9.4.1-1PGDG.rhel7 will be updated
---> Package postgresql94-devel.x86_64 0:9.4.2-2PGDG.rhel7 will be an
update
---> Package postgresql94-libs.x86_64 0:9.4.1-1PGDG.rhel7 will be updated
---> Package postgresql94-libs.x86_64 0:9.4.2-2PGDG.rhel7 will be an update
---> Package postgresql94-server.x86_64 0:9.4.1-1PGDG.rhel7 will be updated
---> Package postgresql94-server.x86_64 0:9.4.2-2PGDG.rhel7 will be an
update
--> Finished Dependency Resolution

Dependencies Resolved

===================================================================================================
Package Arch Version
Repository Size
===================================================================================================
Updating:
postgresql94 x86_64 9.4.2-2PGDG.rhel7
pgdg94 1.0 M
Updating for dependencies:
postgresql94-contrib x86_64 9.4.2-2PGDG.rhel7
pgdg94 609 k
postgresql94-devel x86_64 9.4.2-2PGDG.rhel7
pgdg94 1.6 M
postgresql94-libs x86_64 9.4.2-2PGDG.rhel7
pgdg94 208 k
postgresql94-server x86_64 9.4.2-2PGDG.rhel7
pgdg94 3.9 M

Transaction Summary
===================================================================================================
Upgrade 1 Package (+4 Dependent packages)

Total download size: 7.3 M
Is this ok [y/d/N]: y
Downloading packages:
No Presto metadata available for pgdg94
(1/5): postgresql94-contrib-9.4.2-2PGDG.rhel7.x86_64.rpm
| 609 kB 00:00:00
(2/5): postgresql94-9.4.2-2PGDG.rhel7.x86_64.rpm
| 1.0 MB 00:00:00
(3/5): postgresql94-libs-9.4.2-2PGDG.rhel7.x86_64.rpm
| 208 kB 00:00:00
(4/5): postgresql94-devel-9.4.2-2PGDG.rhel7.x86_64.rpm
| 1.6 MB 00:00:00
(5/5): postgresql94-server-9.4.2-2PGDG.rhel7.x86_64.rpm
| 3.9 MB 00:00:00
---------------------------------------------------------------------------------------------------
Total 5.8 MB/s
| 7.3 MB 00:00:01
Running transaction check
Running transaction test
Transaction test succeeded
Running transaction
Updating : postgresql94-libs-9.4.2-2PGDG.rhel7.x86_64
1/10
Updating : postgresql94-9.4.2-2PGDG.rhel7.x86_64
2/10
Updating : postgresql94-devel-9.4.2-2PGDG.rhel7.x86_64
3/10
Updating : postgresql94-server-9.4.2-2PGDG.rhel7.x86_64
4/10
Updating : postgresql94-contrib-9.4.2-2PGDG.rhel7.x86_64
5/10
Cleanup : postgresql94-contrib-9.4.1-1PGDG.rhel7.x86_64
6/10
Cleanup : postgresql94-server-9.4.1-1PGDG.rhel7.x86_64
7/10
Cleanup : postgresql94-devel-9.4.1-1PGDG.rhel7.x86_64
8/10
Cleanup : postgresql94-9.4.1-1PGDG.rhel7.x86_64
9/10
Cleanup : postgresql94-libs-9.4.1-1PGDG.rhel7.x86_64
10/10
Verifying : postgresql94-devel-9.4.2-2PGDG.rhel7.x86_64
1/10
Verifying : postgresql94-libs-9.4.2-2PGDG.rhel7.x86_64
2/10
Verifying : postgresql94-9.4.2-2PGDG.rhel7.x86_64
3/10
Verifying : postgresql94-server-9.4.2-2PGDG.rhel7.x86_64
4/10
Verifying : postgresql94-contrib-9.4.2-2PGDG.rhel7.x86_64
5/10
Verifying : postgresql94-devel-9.4.1-1PGDG.rhel7.x86_64
6/10
Verifying : postgresql94-server-9.4.1-1PGDG.rhel7.x86_64
7/10
Verifying : postgresql94-9.4.1-1PGDG.rhel7.x86_64
8/10
Verifying : postgresql94-libs-9.4.1-1PGDG.rhel7.x86_64
9/10
Verifying : postgresql94-contrib-9.4.1-1PGDG.rhel7.x86_64
10/10

Updated:
postgresql94.x86_64 0:9.4.2-2PGDG.rhel7

Dependency Updated:
postgresql94-contrib.x86_64 0:9.4.2-2PGDG.rhel7 postgresql94-devel.x86_64
0:9.4.2-2PGDG.rhel7
postgresql94-libs.x86_64 0:9.4.2-2PGDG.rhel7
postgresql94-server.x86_64 0:9.4.2-2PGDG.rhel7

Complete!
root(at)pgtest2:TESTING:cestel> rpm -q postgresql94
postgresql94-9.4.2-2PGDG.rhel7.x86_64

#TRY TO CREATE PG_BASEBACKUP WHILE DESTINATION IS 9.4.2 and SOURCE is 9.4.2
root(at)g1-cestel:DEVELOPMENT:9.4> rpm -q postgresql94
postgresql94-9.4.2-2PGDG.rhel7.x86_64
root(at)g1-cestel:DEVELOPMENT:9.4> service postgresql-9.4 stop
Redirecting to /bin/systemctl stop postgresql-9.4.service
root(at)g1-cestel:DEVELOPMENT:9.4> rm -rf data/*
root(at)g1-cestel:DEVELOPMENT:9.4> su - postgres -c "pg_basebackup -D
/var/lib/pgsql/9.4/data --host=pgtest2 --port=5432 -x -P --username=repl
--password"
Password:
2639858/2639858 kB (100%), 1/1 tablespace
root(at)g1-cestel:DEVELOPMENT:9.4> service postgresql-9.4 restart
Redirecting to /bin/systemctl restart postgresql-9.4.service
Job for postgresql-9.4.service failed. See 'systemctl status
postgresql-9.4.service' and 'journalctl -xn' for details.
root(at)g1-cestel:DEVELOPMENT:9.4> journalctl -xn
-- Logs begin at Thu 2015-05-21 14:13:26 EDT, end at Wed 2015-05-27 12:58:43
EDT. --
May 27 12:57:12 g1-cestel.dev.cmmint.net su[7927]: pam_unix(su-l:session):
session opened for user
May 27 12:58:23 g1-cestel.dev.cmmint.net su[7927]: pam_unix(su-l:session):
session closed for user
May 27 12:58:38 g1-cestel.dev.cmmint.net systemd[1]: Starting PostgreSQL 9.4
database server...
-- Subject: Unit postgresql-9.4.service has begun with start-up
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
--
-- Unit postgresql-9.4.service has begun starting up.
May 27 12:58:38 g1-cestel.dev.cmmint.net pg_ctl[8016]: 2015-05-27 12:58:38
EDT [8021]: [1-1] user=,
May 27 12:58:38 g1-cestel.dev.cmmint.net pg_ctl[8016]: 2015-05-27 12:58:38
EDT [8021]: [2-1] user=,
May 27 12:58:43 g1-cestel.dev.cmmint.net pg_ctl[8016]: pg_ctl: could not
start server
May 27 12:58:43 g1-cestel.dev.cmmint.net pg_ctl[8016]: Examine the log
output.
May 27 12:58:43 g1-cestel.dev.cmmint.net systemd[1]: postgresql-9.4.service:
control process exited
May 27 12:58:43 g1-cestel.dev.cmmint.net systemd[1]: Failed to start
PostgreSQL 9.4 database server
-- Subject: Unit postgresql-9.4.service has failed
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
--
-- Unit postgresql-9.4.service has failed.
--
-- The result is failed.
May 27 12:58:43 g1-cestel.dev.cmmint.net systemd[1]: Unit
postgresql-9.4.service entered failed sta
root(at)g1-cestel:DEVELOPMENT:9.4> cd data/pg_log
root(at)g1-cestel:DEVELOPMENT:pg_log> cat 2015-05-27_1258-postgresql.log
2015-05-27 12:58:38 EDT [8023]: [1-1] user=,db= LOG: database system was
interrupted; last known up at 2015-05-27 13:10:32 EDT
2015-05-27 12:58:39 EDT [8023]: [2-1] user=,db= FATAL: could not open file
"/var/lib/pgsql/9.4/data/pg_log/logrotate": Permission denied
2015-05-27 12:58:39 EDT [8021]: [3-1] user=,db= LOG: startup process (PID
8023) exited with exit code 1
2015-05-27 12:58:39 EDT [8021]: [4-1] user=,db= LOG: aborting startup due
to startup process failure
root(at)g1-cestel:DEVELOPMENT:pg_log> rm logrotate
rm: remove regular empty file ‘logrotate’? y
root(at)g1-cestel:DEVELOPMENT:pg_log> service postgresql-9.4 restart
Redirecting to /bin/systemctl restart postgresql-9.4.service
Job for postgresql-9.4.service failed. See 'systemctl status
postgresql-9.4.service' and 'journalctl -xn' for details.
root(at)g1-cestel:DEVELOPMENT:pg_log> journalctl -xn
-- Logs begin at Thu 2015-05-21 14:13:26 EDT, end at Wed 2015-05-27 13:01:01
EDT. --
May 27 13:00:57 g1-cestel.dev.cmmint.net pg_ctl[8062]: pg_ctl: could not
start server
May 27 13:00:57 g1-cestel.dev.cmmint.net pg_ctl[8062]: Examine the log
output.
May 27 13:00:57 g1-cestel.dev.cmmint.net systemd[1]: postgresql-9.4.service:
control process exited
May 27 13:00:57 g1-cestel.dev.cmmint.net systemd[1]: Failed to start
PostgreSQL 9.4 database server
-- Subject: Unit postgresql-9.4.service has failed
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
--
-- Unit postgresql-9.4.service has failed.
--
-- The result is failed.
May 27 13:00:57 g1-cestel.dev.cmmint.net systemd[1]: Unit
postgresql-9.4.service entered failed sta
May 27 13:01:01 g1-cestel.dev.cmmint.net CROND[8071]: (root) CMD (run-parts
/etc/cron.hourly)
May 27 13:01:01 g1-cestel.dev.cmmint.net run-parts(/etc/cron.hourly)[8074]:
starting 0anacron
May 27 13:01:01 g1-cestel.dev.cmmint.net run-parts(/etc/cron.hourly)[8080]:
finished 0anacron
May 27 13:01:01 g1-cestel.dev.cmmint.net run-parts(/etc/cron.hourly)[8082]:
starting 0yum-hourly.cr
May 27 13:01:01 g1-cestel.dev.cmmint.net run-parts(/etc/cron.hourly)[8086]:
finished 0yum-hourly.cr
root(at)g1-cestel:DEVELOPMENT:pg_log> cat 2015-05-27_1300-postgresql.log
2015-05-27 13:00:53 EDT [8069]: [1-1] user=,db= LOG: database system was
interrupted; last known up at 2015-05-27 13:10:32 EDT
2015-05-27 13:00:53 EDT [8069]: [2-1] user=,db= FATAL: could not open file
"/var/lib/pgsql/9.4/data/server.crt": Permission denied
2015-05-27 13:00:53 EDT [8067]: [3-1] user=,db= LOG: startup process (PID
8069) exited with exit code 1
2015-05-27 13:00:53 EDT [8067]: [4-1] user=,db= LOG: aborting startup due
to startup process failure
root(at)g1-cestel:DEVELOPMENT:pg_log> ls -lsa
/var/lib/pgsql/9.4/data/server.crt
4 -r-------- 1 postgres postgres 2297 May 27 12:58
/var/lib/pgsql/9.4/data/server.crt
root(at)g1-cestel:DEVELOPMENT:pg_log> chmod +r
/var/lib/pgsql/9.4/data/server.crt
root(at)g1-cestel:DEVELOPMENT:pg_log> ls -lsa
/var/lib/pgsql/9.4/data/server.crt
4 -r--r--r-- 1 postgres postgres 2297 May 27 12:58
/var/lib/pgsql/9.4/data/server.crt
root(at)g1-cestel:DEVELOPMENT:pg_log> service postgresql-9.4 restart
Redirecting to /bin/systemctl restart postgresql-9.4.service
Job for postgresql-9.4.service failed. See 'systemctl status
postgresql-9.4.service' and 'journalctl -xn' for details.
root(at)g1-cestel:DEVELOPMENT:pg_log> cat 2015-05-27_1302-postgresql.log
2015-05-27 13:02:45 EDT [8150]: [1-1] user=,db= LOG: database system was
interrupted; last known up at 2015-05-27 13:10:32 EDT
2015-05-27 13:02:46 EDT [8150]: [2-1] user=,db= FATAL: could not open file
"/var/lib/pgsql/9.4/data/server.crt": Permission denied
2015-05-27 13:02:46 EDT [8148]: [3-1] user=,db= LOG: startup process (PID
8150) exited with exit code 1
2015-05-27 13:02:46 EDT [8148]: [4-1] user=,db= LOG: aborting startup due
to startup process failure

DOWNGRADE DESTINATION TO 9.4.1 WHILE SOURCE IS 9.4.2 AND RUN PG_BASEBACKUP
root(at)g1-cestel:DEVELOPMENT:pg_log> yum downgrade postgresql94
postgresql94-devel postgresql94-libs postgresql94-server
postgresql94-contrib
Loaded plugins: langpacks, product-id, rhnplugin, subscription-manager
This system is receiving updates from RHN Classic or Red Hat Satellite.
Resolving Dependencies
--> Running transaction check
---> Package postgresql94.x86_64 0:9.4.2-1PGDG.rhel7 will be a downgrade
---> Package postgresql94.x86_64 0:9.4.2-2PGDG.rhel7 will be erased
---> Package postgresql94-contrib.x86_64 0:9.4.2-1PGDG.rhel7 will be a
downgrade
---> Package postgresql94-contrib.x86_64 0:9.4.2-2PGDG.rhel7 will be erased
---> Package postgresql94-devel.x86_64 0:9.4.2-1PGDG.rhel7 will be a
downgrade
---> Package postgresql94-devel.x86_64 0:9.4.2-2PGDG.rhel7 will be erased
---> Package postgresql94-libs.x86_64 0:9.4.2-1PGDG.rhel7 will be a
downgrade
---> Package postgresql94-libs.x86_64 0:9.4.2-2PGDG.rhel7 will be erased
---> Package postgresql94-server.x86_64 0:9.4.2-1PGDG.rhel7 will be a
downgrade
---> Package postgresql94-server.x86_64 0:9.4.2-2PGDG.rhel7 will be erased
--> Finished Dependency Resolution

Dependencies Resolved

===================================================================================================
Package Arch Version
Repository Size
===================================================================================================
Downgrading:
postgresql94 x86_64 9.4.2-1PGDG.rhel7
yum.postgresql.org 1.0 M
postgresql94-contrib x86_64 9.4.2-1PGDG.rhel7
yum.postgresql.org 609 k
postgresql94-devel x86_64 9.4.2-1PGDG.rhel7
yum.postgresql.org 1.6 M
postgresql94-libs x86_64 9.4.2-1PGDG.rhel7
yum.postgresql.org 208 k
postgresql94-server x86_64 9.4.2-1PGDG.rhel7
yum.postgresql.org 3.9 M

Transaction Summary
===================================================================================================
Downgrade 5 Packages

Total download size: 7.3 M
Is this ok [y/d/N]: y
Downloading packages:
(1/5): postgresql94-contrib-9.4.2-1PGDG.rhel7.x86_64.rpm
| 609 kB 00:00:00
(2/5): postgresql94-9.4.2-1PGDG.rhel7.x86_64.rpm
| 1.0 MB 00:00:00
(3/5): postgresql94-libs-9.4.2-1PGDG.rhel7.x86_64.rpm
| 208 kB 00:00:00
(4/5): postgresql94-devel-9.4.2-1PGDG.rhel7.x86_64.rpm
| 1.6 MB 00:00:00
(5/5): postgresql94-server-9.4.2-1PGDG.rhel7.x86_64.rpm
| 3.9 MB 00:00:01
---------------------------------------------------------------------------------------------------
Total 3.4 MB/s
| 7.3 MB 00:00:02
Running transaction check
Running transaction test
Transaction test succeeded
Running transaction
Installing : postgresql94-libs-9.4.2-1PGDG.rhel7.x86_64
1/10
Installing : postgresql94-9.4.2-1PGDG.rhel7.x86_64
2/10
Installing : postgresql94-contrib-9.4.2-1PGDG.rhel7.x86_64
3/10
Installing : postgresql94-server-9.4.2-1PGDG.rhel7.x86_64
4/10
Installing : postgresql94-devel-9.4.2-1PGDG.rhel7.x86_64
5/10
Cleanup : postgresql94-contrib-9.4.2-2PGDG.rhel7.x86_64
6/10
Cleanup : postgresql94-server-9.4.2-2PGDG.rhel7.x86_64
7/10
Cleanup : postgresql94-devel-9.4.2-2PGDG.rhel7.x86_64
8/10
Cleanup : postgresql94-9.4.2-2PGDG.rhel7.x86_64
9/10
Cleanup : postgresql94-libs-9.4.2-2PGDG.rhel7.x86_64
10/10
Verifying : postgresql94-libs-9.4.2-1PGDG.rhel7.x86_64
1/10
Verifying : postgresql94-contrib-9.4.2-1PGDG.rhel7.x86_64
2/10
Verifying : postgresql94-9.4.2-1PGDG.rhel7.x86_64
3/10
Verifying : postgresql94-server-9.4.2-1PGDG.rhel7.x86_64
4/10
Verifying : postgresql94-devel-9.4.2-1PGDG.rhel7.x86_64
5/10
Verifying : postgresql94-devel-9.4.2-2PGDG.rhel7.x86_64
6/10
Verifying : postgresql94-server-9.4.2-2PGDG.rhel7.x86_64
7/10
Verifying : postgresql94-9.4.2-2PGDG.rhel7.x86_64
8/10
Verifying : postgresql94-contrib-9.4.2-2PGDG.rhel7.x86_64
9/10
Verifying : postgresql94-libs-9.4.2-2PGDG.rhel7.x86_64
10/10

Removed:
postgresql94.x86_64 0:9.4.2-2PGDG.rhel7
postgresql94-contrib.x86_64 0:9.4.2-2PGDG.rhel7
postgresql94-devel.x86_64 0:9.4.2-2PGDG.rhel7 postgresql94-libs.x86_64
0:9.4.2-2PGDG.rhel7
postgresql94-server.x86_64 0:9.4.2-2PGDG.rhel7

Installed:
postgresql94.x86_64 0:9.4.2-1PGDG.rhel7
postgresql94-contrib.x86_64 0:9.4.2-1PGDG.rhel7
postgresql94-devel.x86_64 0:9.4.2-1PGDG.rhel7 postgresql94-libs.x86_64
0:9.4.2-1PGDG.rhel7
postgresql94-server.x86_64 0:9.4.2-1PGDG.rhel7

Complete!
root(at)g1-cestel:DEVELOPMENT:pg_log> yum downgrade postgresql94
postgresql94-devel postgresql94-libs postgresql94-server
postgresql94-contrib
Loaded plugins: langpacks, product-id, rhnplugin, subscription-manager
This system is receiving updates from RHN Classic or Red Hat Satellite.
Resolving Dependencies
--> Running transaction check
---> Package postgresql94.x86_64 0:9.4.1-1PGDG.rhel7 will be a downgrade
---> Package postgresql94.x86_64 0:9.4.2-1PGDG.rhel7 will be erased
---> Package postgresql94-contrib.x86_64 0:9.4.1-1PGDG.rhel7 will be a
downgrade
---> Package postgresql94-contrib.x86_64 0:9.4.2-1PGDG.rhel7 will be erased
---> Package postgresql94-devel.x86_64 0:9.4.1-1PGDG.rhel7 will be a
downgrade
---> Package postgresql94-devel.x86_64 0:9.4.2-1PGDG.rhel7 will be erased
---> Package postgresql94-libs.x86_64 0:9.4.1-1PGDG.rhel7 will be a
downgrade
---> Package postgresql94-libs.x86_64 0:9.4.2-1PGDG.rhel7 will be erased
---> Package postgresql94-server.x86_64 0:9.4.1-1PGDG.rhel7 will be a
downgrade
---> Package postgresql94-server.x86_64 0:9.4.2-1PGDG.rhel7 will be erased
--> Finished Dependency Resolution

Dependencies Resolved

===================================================================================================
Package Arch Version
Repository Size
===================================================================================================
Downgrading:
postgresql94 x86_64 9.4.1-1PGDG.rhel7
yum.postgresql.org 1.0 M
postgresql94-contrib x86_64 9.4.1-1PGDG.rhel7
yum.postgresql.org 602 k
postgresql94-devel x86_64 9.4.1-1PGDG.rhel7
yum.postgresql.org 1.6 M
postgresql94-libs x86_64 9.4.1-1PGDG.rhel7
yum.postgresql.org 202 k
postgresql94-server x86_64 9.4.1-1PGDG.rhel7
yum.postgresql.org 3.9 M

Transaction Summary
===================================================================================================
Downgrade 5 Packages

Total download size: 7.3 M
Is this ok [y/d/N]: y
Downloading packages:
(1/5): postgresql94-contrib-9.4.1-1PGDG.rhel7.x86_64.rpm
| 602 kB 00:00:00
(2/5): postgresql94-devel-9.4.1-1PGDG.rhel7.x86_64.rpm
| 1.6 MB 00:00:00
(3/5): postgresql94-libs-9.4.1-1PGDG.rhel7.x86_64.rpm
| 202 kB 00:00:00
(4/5): postgresql94-9.4.1-1PGDG.rhel7.x86_64.rpm
| 1.0 MB 00:00:01
(5/5): postgresql94-server-9.4.1-1PGDG.rhel7.x86_64.rpm
| 3.9 MB 00:00:00
---------------------------------------------------------------------------------------------------
Total 3.2 MB/s
| 7.3 MB 00:00:02
Running transaction check
Running transaction test
Transaction test succeeded
Running transaction
Installing : postgresql94-libs-9.4.1-1PGDG.rhel7.x86_64
1/10
Installing : postgresql94-9.4.1-1PGDG.rhel7.x86_64
2/10
Installing : postgresql94-devel-9.4.1-1PGDG.rhel7.x86_64
3/10
Installing : postgresql94-contrib-9.4.1-1PGDG.rhel7.x86_64
4/10
Installing : postgresql94-server-9.4.1-1PGDG.rhel7.x86_64
5/10
Cleanup : postgresql94-server-9.4.2-1PGDG.rhel7.x86_64
6/10
Cleanup : postgresql94-contrib-9.4.2-1PGDG.rhel7.x86_64
7/10
Cleanup : postgresql94-devel-9.4.2-1PGDG.rhel7.x86_64
8/10
Cleanup : postgresql94-9.4.2-1PGDG.rhel7.x86_64
9/10
Cleanup : postgresql94-libs-9.4.2-1PGDG.rhel7.x86_64
10/10
Verifying : postgresql94-devel-9.4.1-1PGDG.rhel7.x86_64
1/10
Verifying : postgresql94-contrib-9.4.1-1PGDG.rhel7.x86_64
2/10
Verifying : postgresql94-server-9.4.1-1PGDG.rhel7.x86_64
3/10
Verifying : postgresql94-9.4.1-1PGDG.rhel7.x86_64
4/10
Verifying : postgresql94-libs-9.4.1-1PGDG.rhel7.x86_64
5/10
Verifying : postgresql94-libs-9.4.2-1PGDG.rhel7.x86_64
6/10
Verifying : postgresql94-9.4.2-1PGDG.rhel7.x86_64
7/10
Verifying : postgresql94-server-9.4.2-1PGDG.rhel7.x86_64
8/10
Verifying : postgresql94-contrib-9.4.2-1PGDG.rhel7.x86_64
9/10
Verifying : postgresql94-devel-9.4.2-1PGDG.rhel7.x86_64
10/10

Removed:
postgresql94.x86_64 0:9.4.2-1PGDG.rhel7
postgresql94-contrib.x86_64 0:9.4.2-1PGDG.rhel7
postgresql94-devel.x86_64 0:9.4.2-1PGDG.rhel7 postgresql94-libs.x86_64
0:9.4.2-1PGDG.rhel7
postgresql94-server.x86_64 0:9.4.2-1PGDG.rhel7

Installed:
postgresql94.x86_64 0:9.4.1-1PGDG.rhel7
postgresql94-contrib.x86_64 0:9.4.1-1PGDG.rhel7
postgresql94-devel.x86_64 0:9.4.1-1PGDG.rhel7 postgresql94-libs.x86_64
0:9.4.1-1PGDG.rhel7
postgresql94-server.x86_64 0:9.4.1-1PGDG.rhel7

Complete!
root(at)g1-cestel:DEVELOPMENT:9.4> rpm -q postgresql94
postgresql94-9.4.1-1PGDG.rhel7.x86_64
ooot(at)g1-cestel:DEVELOPMENT:pg_log> cd ../../
root(at)g1-cestel:DEVELOPMENT:9.4> rm -rf data/*
root(at)g1-cestel:DEVELOPMENT:9.4> su - postgres -c "pg_basebackup -D
/var/lib/pgsql/9.4/data --host=pgtest2 --port=5432 -x -P --username=repl
--password"
Password:
2639908/2639908 kB (100%), 1/1 tablespace
root(at)g1-cestel:DEVELOPMENT:9.4> service postgresql-9.4 restart
Redirecting to /bin/systemctl restart postgresql-9.4.service
root(at)g1-cestel:DEVELOPMENT:9.4> su postgres -c psql
psql (9.4.1)
Type "help" for help.

postgres=# \q