Re: Postgres Crash Issue

On Tue, Mar 15, 2022 at 07:20:57AM +0000, Sankar, Uma (Uma) wrote:
> Hi All,
>
> We have a database crash issue last Friday and it's a patroni HA-based
> Postgres database running, we have checked the pg logs and it shows
> error as shared memory corruptions. Can someone please check the
> attached logs and share your suggestions and feedback on the crash
> issues. Thanks in advance.

It wasn't crash.

Logs show:

#v+
2022-03-11 08:58:42.956 UTC [27799] LOG: received immediate shutdown request
2022-03-11 08:58:42.956 UTC [17115] WARNING: terminating connection because of crash of another server process
2022-03-11 08:58:42.956 UTC [17115] DETAIL: The postmaster has commanded this server process to roll back the current transaction and exit, because another server process exited abnormally and possibly corrupted shared memory.
2022-03-11 08:58:42.956 UTC [17115] HINT: In a moment you should be able to reconnect to the database and repeat your command.
2022-03-11 08:58:42.957 UTC [14618] WARNING: terminating connection because of crash of another server process
2022-03-11 08:58:42.957 UTC [14618] DETAIL: The postmaster has commanded this server process to roll back the current transaction and exit, because another server process exited abnormally and possibly corrupted shared memory.
2022-03-11 08:58:42.957 UTC [14618] HINT: In a moment you should be able to reconnect to the database and repeat your command.
2022-03-11 08:58:42.957 UTC [16396] WARNING: terminating connection because of crash of another server process
2022-03-11 08:58:42.957 UTC [16396] DETAIL: The postmaster has commanded this server process to roll back the current transaction and exit, because another server process exited abnormally and possibly corrupted shared memory.
2022-03-11 08:58:42.957 UTC [16396] HINT: In a moment you should be able to reconnect to the database and repeat your command.
2022-03-11 08:58:42.957 UTC [12873] WARNING: terminating connection because of crash of another server process
2022-03-11 08:58:42.957 UTC [12873] DETAIL: The postmaster has commanded this server process to roll back the current transaction and exit, because another server process exited abnormally and possibly corrupted shared memory.
2022-03-11 08:58:42.957 UTC [12873] HINT: In a moment you should be able to reconnect to the database and repeat your command.
2022-03-11 08:58:42.958 UTC [16564] WARNING: terminating connection because of crash of another server process
2022-03-11 08:58:42.958 UTC [16564] DETAIL: The postmaster has commanded this server process to roll back the current transaction and exit, because another server process exited abnormally and possibly corrupted shared memory.
2022-03-11 08:58:42.958 UTC [16564] HINT: In a moment you should be able to reconnect to the database and repeat your command.
#v-

and

#v+
2022-03-11 08:58:45.969 UTC [17354] LOG: consistent recovery state reached at A/E1836620
2022-03-11 08:58:45.969 UTC [17347] LOG: database system is ready to accept read only connections
2022-03-11 08:59:01.234 UTC [17347] LOG: received fast shutdown request
2022-03-11 08:59:01.234 UTC [17347] LOG: aborting any active transactions
2022-03-11 08:59:01.234 UTC [17412] FATAL: terminating connection due to administrator command
2022-03-11 08:59:01.234 UTC [17415] FATAL: terminating connection due to administrator command
2022-03-11 08:59:01.234 UTC [17393] FATAL: terminating connection due to administrator command
2022-03-11 08:59:01.235 UTC [17399] FATAL: terminating connection due to administrator command
2022-03-11 08:59:01.235 UTC [17408] FATAL: terminating connection due to administrator command
2022-03-11 08:59:01.236 UTC [17394] FATAL: terminating connection due to administrator command
2022-03-11 08:59:01.236 UTC [17389] FATAL: terminating connection due to administrator command
2022-03-11 08:59:01.237 UTC [17400] FATAL: terminating connection due to administrator command
2022-03-11 08:59:01.237 UTC [17378] FATAL: terminating connection due to administrator command
2022-03-11 08:59:01.238 UTC [17362] LOG: shutting down
2022-03-11 08:59:01.254 UTC [17347] LOG: database system is shut down
2022-03-11 08:59:01.419 UTC [17444] LOG: database system was shut down in recovery at 2022-03-11 08:59:01 UTC
2022-03-11 08:59:01.419 UTC [17444] LOG: entering standby mode
2022-03-11 08:59:01.420 UTC [17444] LOG: redo starts at A/E17CB690
#v-

Something/someone stopped the pg, twice, ~ 20 seconds apart.
What/how/why - can't tell, but "received ... shutdown request" is not
crash. It is someone/something doing "pg_ctl stop"

depesz