RE: Postgres 10.7 Systemd Startup Issue

Thanks for reaching out Francisco. Here's some relevant information related to your questions.

Here's the service we're running (removed superfluous stuff like comments):

[Unit]
Description=PostgreSQL 10 database server
Documentation=https://www.postgresql.org/docs/10/static/
After=syslog.target
After=network.target

[Service]
Type=forking
User=postgresdbad
Group=postgresdba
Environment=PGDATA=/hostname/pg/pgdba1dbal/data
OOMScoreAdjust=-1000
Environment=PG_OOM_ADJUST_FILE=/proc/self/oom_score_adj
Environment=PG_OOM_ADJUST_VALUE=0
ExecStart=/hostname/pg/PostgreSQL-10.7/bin/postgres -D ${PGDATA}
ExecReload=/bin/kill -HUP $MAINPID
KillMode=mixed
KillSignal=SIGINT
TimeoutSec=0

[Install]
WantedBy=multi-user.target

Here's the actual status:

● postgres.service - PostgreSQL 10 database server
Loaded: loaded (/etc/systemd/system/postgres.service; enabled; vendor preset: disabled)
Active: activating (start) since Thu 2019-06-06 14:00:39 EDT; 23s ago
Docs: https://www.postgresql.org/docs/10/static/
Main PID: 4094 (code=exited, status=2); : 4382 (postgres)
Tasks: 9
CGroup: /system.slice/postgres.service
├─4382 /hostname/pg/PostgreSQL-10.7/bin/postgres -D /hostname/pg/postgres/data
├─4385 postgres: logger process
├─4387 postgres: checkpointer process
├─4388 postgres: writer process
├─4389 postgres: wal writer process
├─4390 postgres: autovacuum launcher process
├─4391 postgres: archiver process
├─4392 postgres: stats collector process
└─4393 postgres: bgworker: logical replication launcher

Jun 06 14:00:39 pgdbadevbal800 systemd[1]: Starting PostgreSQL 10 database server...
Jun 06 14:00:39 pgdbadevbal800 postgres[4382]: 2019-06-06 14:00:39.692 EDT [4382] LOG: listening on IPv4 address "##.###.###.###", port 5432
Jun 06 14:00:39 pgdbadevbal800 postgres[4382]: 2019-06-06 14:00:39.696 EDT [4382] LOG: listening on Unix socket "/tmp/.s.PGSQL.5432"
Jun 06 14:00:39 pgdbadevbal800 postgres[4382]: 2019-06-06 14:00:39.704 EDT [4382] LOG: redirecting log output to logging collector process
Jun 06 14:00:39 pgdbadevbal800 postgres[4382]: 2019-06-06 14:00:39.704 EDT [4382] HINT: Future log output will appear in directory "/hostname/pg/postgres/log".

It appears to fork successfully however the state still shows as "activating" and remains like that. We have another box in our environment, same OS but is running postgres 10.3-1PGDG.sles12 official package. The 10.7 in the example above is from a tarball. Both hosts are using near identical systemd service files. The other box will start up and reach an "Active (running)" state.

Other related information:
- systemd version 228-150.29.1 (on both hosts)
- os SUSE Linux Enterprise Server 12 SP3 (on both)
- I do run a systemctl daemon-reload after any changes to service files.

Thank you again for reaching out!

-Kevin Kelly

-----Original Message-----
From: Francisco Olarte [mailto:folarte(at)peoplecall(dot)com]
Sent: Thursday, June 06, 2019 1:44 PM
To: Kelly, Kevin <Kevin(dot)Kelly(at)msx(dot)bala(dot)susq(dot)com>
Cc: pgsql-general(at)postgresql(dot)org; Lu, Dan <Dan(dot)Lu(at)msx(dot)bala(dot)susq(dot)com>
Subject: Re: Postgres 10.7 Systemd Startup Issue

On Thu, Jun 6, 2019 at 6:57 PM Kelly, Kevin <Kevin(dot)Kelly(at)sig(dot)com> wrote:
> We’re attempting to launch postgres via systemd and noticing that when invoking via systemctl start postgres.service the prompt never returns. If we switch to another tty and check the status it shows as:
> Active: activating (start) since Thu 2019-06-06 09:36:32 EDT; 12min
> ago If we change the type from notify to forking we see the same behavior. The instance seems to be up and running just fine, we just never see the active (running) status as we have come to expect.

Are you sure you have the postgres.service correctly configured? ( or you could post the ExecStart/Type config, or the whole service file ).

The type tells systemd how to know the service has finished starting and is running, notify means it does systemd integration via socket, with IIRC postgres does not. Forking means the usual server approach, server detachs and after ExecStart returns it is considered running.
Simple is for not detaching.

Your symptom seem to be caused by either using a execstart command which does not detach ( so forking waits forever for initialisation to finish ) or for not having done daemon-reload to systemd ( so you are still in notify mode and it is waiting forever for postgres to tell him it is ready ).

> We’re running on top of a stock SLES12 SP3. Could someone assist in tracking down why this service isn’t transitioning to the active state?

Try posting some log, the service file ( to know which command you use ), maybe a ps trace, but I would go for checking the two problems above first. My be t is on the second one ( and if it is that you'll have the surprise of it working after machine reboot ).

Bear in mind many of us, althoguh wanting to help, do not even know what "stock SLES12 SP3" is, so no idea on systemd version and/or service files content.

Francisco Olarte.

________________________________

IMPORTANT: The information contained in this email and/or its attachments is confidential. If you are not the intended recipient, please notify the sender immediately by reply and immediately delete this message and all its attachments. Any review, use, reproduction, disclosure or dissemination of this message or any attachment by an unintended recipient is strictly prohibited. Neither this message nor any attachment is intended as or should be construed as an offer, solicitation or recommendation to buy or sell any security or other financial instrument. Neither the sender, his or her employer nor any of their respective affiliates makes any warranties as to the completeness or accuracy of any of the information contained herein or that this message or any of its attachments is free of viruses.