RE: RHEL 7 (systemd) reboot

Sorry, I wasn't clear in the prior posts.

The stop script is running during reboot. The problem is the database is not reachable when the stop script runs. The ctmdist server shut down is as follows:
Stop control-m application
Stop control-m configuration agent
Stop database

As you can see the intent is for the database to be shut down after the product.

But as you noticed from /var/log/message the stop_ctmlinux_server.sh script is running but unable to execute the update query.

I created the following Service definition and scripts that follow -- note there are 2 datacenters (ctmdist, ctmlinux) that have comparable scripts so I have only included one set:

[root(at)kccontrolmt01 ~]# cat ControlM_Shutdown.service
[Unit]
Description=Run mycommand at shutdown
Requires=network.target CTM_Postgre.service
DefaultDependencies=no
Before=shutdown.target reboot.target

[Service]
Type=oneshot
RemainAfterExit=true
ExecStart=/bin/true
ExecStop=/root/scripts/control-m_shutdown.sh

[Install]
WantedBy=multi-user.target

[root(at)kccontrolmt01 ~]# cat /root/scripts/control-m_shutdown.sh
#!/bin/sh
# Shutdown any running Control-M services
STATUS=$(/usr/bin/systemctl is-active CTMLinux_Server.service)
if [ ${STATUS} == "active" ]; then
/usr/bin/systemctl stop CTMLinux_Server.service
fi

STATUS=$(/usr/bin/systemctl is-active CTMDist_Server.service)
if [ ${STATUS} == "active" ]; then
/usr/bin/systemctl stop CTMDist_Server.service
fi

STATUS=$(/usr/bin/systemctl is-active EnterpriseManager.service)
if [ ${STATUS} == "active" ]; then
/usr/bin/systemctl stop EnterpriseManager.service
fi
exit 0

#!/bin/bash

# stop CONTROL-M
if [ -f /data00/ctmlinux/ctm_server/scripts/shut_ctm ]; then
echo "Stopping CONTROL-M application"
/data00/ctmlinux/ctm_server/scripts/shut_ctm
fi

# stop CONTROL-M Configuration Agent
if [ -f /data00/ctmlinux/ctm_server/scripts/shut_ca ]; then
echo "Stopping CONTROL-M Server Configuration Agent"
/data00/ctmlinux/ctm_server/scripts/shut_ca
fi

# stop database
/data00/ctmlinux/ctm_server/scripts/dbversion
if [ $? -ne 0 ] ; then
echo "SQL Server is already stopped "
else
if [ -f /data00/ctmlinux/ctm_server/scripts/shutdb ]; then
echo "Stopping SQL server for CONTROL-M"
/data00/ctmlinux/ctm_server/scripts/shutdb
fi
fi

exit 0

-----Original Message-----
From: Adrian Klaver <adrian(dot)klaver(at)aklaver(dot)com>
Sent: Wednesday, October 10, 2018 8:25 AM
To: Bryce Pepper <BPepper(at)KCSouthern(dot)com>; pgsql-general(at)lists(dot)postgresql(dot)org
Subject: Re: RHEL 7 (systemd) reboot

This email originated from outside the company. Please use caution when opening attachments or clicking on links. If you suspect this to be a phishing attempt, please report via PhishAlarm.
________________________________

On 10/10/18 5:32 AM, Bryce Pepper wrote:
> Adrian,
> Thanks for the inquiry. The function (db_execute_sql) is coming from a vendor (BMC) product called Control-M. It is a scheduling product.
> The tmp file is deleted before I can see its contents but I believe it is trying to update some columns in the CMS_SYSPRM table.
> I also think the postgresql instance is already stopped and hence why the db_execute fails. I will try to modify the vendor function to save off the contents of the query.

Alright, I'm confused. In your earlier post you said the stop script is not running. Yet here it is, just not at the right time. I think a more detailed explanation is needed:

1) The stop script you are concerned about is a systemd script, one that you created or system provided?

2) What is the shutdown service you refer to?

3) Is there a separate shutdown script for the Control-M product?

4) What do you expect to happen vs what is happening?

>
> Bryce
>
> p.s. Do you know of any verbose logging that could be turned on to catch when pgsql is being terminated?
>
>
> -----Original Message-----
> From: Adrian Klaver <adrian(dot)klaver(at)aklaver(dot)com>
> Sent: Tuesday, October 09, 2018 7:39 PM
> To: Bryce Pepper <BPepper(at)KCSouthern(dot)com>;
> pgsql-general(at)lists(dot)postgresql(dot)org
> Subject: Re: RHEL 7 (systemd) reboot
>
> This email originated from outside the company. Please use caution when opening attachments or clicking on links. If you suspect this to be a phishing attempt, please report via PhishAlarm.
> ________________________________
>
> On 10/9/18 11:06 AM, Bryce Pepper wrote:
>> I am running three instances (under different users) on a RHEL 7
>> server to support a vendor product.
>>
>> In the defined services, the start & stop scripts work fine when
>> invoked with systemctl {start|stop} whatever.service but we have
>> automated monthly patching which does a reboot.
>>
>> Looking in /var/log/messages and the stop scripts do not get invoked
>> on reboot, therefore I created a new shutdown service as described
>> here <https://unix.stackexchange.com/questions/211924/effect-of-reboot-signal-on-systemd-service-state>.
>>
>> It appears that PostGreSQL is receiving a signal from somewhere prior
>> to my script running.
>>
>
>>
>> The database must be available for the product to shut down in a
>> consistent state.
>>
>> I am open to suggestions.
>
> What is the below doing or coming from?:
>
> db_execute_sql failed while processing
> /data00/ctmlinux/ctm_server/tmp/upd_CMS_SYSPRM_29448.
>
>>
>> Thanks,
>>
>> Bryce
>>
>> *Bryce Pepper*
>>
>> Sr. Unix Applications Systems Engineer
>>
>> *The Kansas City Southern Railway Company *
>>
>> 114 West 11^th Street |  Kansas City, MO 64105
>>
>> Office: 816.983.1512
>>
>> Email: bpepper(at)kcsouthern(dot)com <mailto:bpepper(at)kcsouthern(dot)com>
>>
>
>
> --
> Adrian Klaver
> adrian(dot)klaver(at)aklaver(dot)com
>

--
Adrian Klaver
adrian(dot)klaver(at)aklaver(dot)com