soft lockup - CPU#16 stuck for 3124s! [postmaster:2273]

We have a PostgreSQL 15.1 server in production at a customer for some
weeks (migrated from an older version) on SuSE SLES 15.

The customer is facing machine locks and before the Linux server does
not respond any more (not even on SSH, only power-cycle reset helps to
get it back), short before the fault a lot of messages are in
/var/log/messages of the content:

# grep watchdog: /var/log/messages
...
2024-03-22T13:11:32.056154+01:00 sunrise kernel: [327844.313048][ C25] watchdog: BUG: soft lockup - CPU#25 stuck for 3069s! [migration/25:166]
2024-03-22T13:12:28.056244+01:00 sunrise kernel: [327900.310267][ C16] watchdog: BUG: soft lockup - CPU#16 stuck for 3124s! [postmaster:2273]
2024-03-22T13:12:28.056340+01:00 sunrise kernel: [327900.311052][ C25] watchdog: BUG: soft lockup - CPU#25 stuck for 3121s! [migration/25:166]

Not all related to postmaster, but some of them. The server is in
principle idle, has a lot of CPUs and 32 GByte memory. To the PostgreSQL
server connect around 100 PostgreSQL clients, most of them by ESQL/C and
on localhost.

Looking around, I detected today that the WAL archiving was configured
wrong, leading to messages like (sorry for the German, but you will get
the meaning):

2024-03-22 13:11:50.838 CET [2630] LOG: Archivbefehl ist fehlgeschlagen mit Statuscode 1
2024-03-22 13:11:50.838 CET [2630] DETAIL: Der fehlgeschlagene Archivbefehl war: test ! -f /data/postgresql151/wal_archive/000000010000000000000001 && cp pg_wal/000000010000000000000001 /data/postgresql151/wal_archive/000000010000000000000001
cp: reguläre Datei '/data/postgresql151/wal_archive/000000010000000000000001' kann nicht angelegt werden: Datei oder Verzeichnis nicht gefunden
2024-03-22 13:11:51.842 CET [2630] LOG: Archivbefehl ist fehlgeschlagen mit Statuscode 1
2024-03-22 13:11:51.842 CET [2630] DETAIL: Der fehlgeschlagene Archivbefehl war: test ! -f /data/postgresql151/wal_archive/000000010000000000000001 && cp pg_wal/000000010000000000000001 /data/postgresql151/wal_archive/000000010000000000000001

The problem was that the directory /data/postgresql151/wal_archive was just
not created (and this for two weeks in production). Since it is now created
and also the backup of the WAL from there is in place, the problem of
the locks went away.

Any chance that the problem of the Pos server not being able to copy the
WALs could have caused the locks? Just to make sure that we hit the beast.

matthias

--
Matthias Apitz, ✉ guru(at)unixarea(dot)de, http://www.unixarea.de/ +49-176-38902045
Public GnuPG key: http://www.unixarea.de/key.pub