Possible Corrputed shared memory

From: James Sebastian <james(dot)sebastian(at)gmail(dot)com>
To: pgsql-novice(at)postgresql(dot)org
Subject: Possible Corrputed shared memory
Date: 2015-08-01 08:41:45
Message-ID: CA+ehAmHAjTmLcxcVGKrGgM30T6fpprSo8RXsn6T57m2s1E8WLQ@mail.gmail.com
Views: Raw Message | Whole Thread | Download mbox | Resend email
Thread:
Lists: pgsql-novice

Dear all,

I am novice to postgres admin world.

I am using postgresl 9.1.11 on Ubutu 12.04 and this is used by application
server running on tomcat.

The application slowed down considerably and the postgresql-9.1-main.log
reported pgstat wait timeout as below.

2015-07-30 09:38:20 IST WARNING: pgstat wait timeout
2015-07-30 09:50:20 IST WARNING: pgstat wait timeout
2015-07-30 09:50:26 IST WARNING: pgstat wait timeout
2015-07-30 10:00:55 IST WARNING: pgstat wait timeout
2015-07-30 10:04:51 IST WARNING: pgstat wait timeout
2015-07-30 10:06:51 IST WARNING: pgstat wait timeout
2015-07-30 10:10:21 IST WARNING: pgstat wait timeout
2015-07-30 10:12:21 IST WARNING: pgstat wait timeout

I attempted application stop and database shutdown using init scripts
followed by start.
The database failed to restart with the following error.

2015-07-30 14:14:17 IST LOG: received immediate shutdown request
2015-07-30 14:14:17 IST WARNING: terminating connection because of crash
of another server process
2015-07-30 14:14:17 IST DETAIL: The postmaster has commanded this server
process to roll back the current transaction and exit, because another
server process exited abnormally and possibly corrupted shared memory.
2015-07-30 14:14:17 IST HINT: In a moment you should be able to reconnect
to the database and repeat your command.
2015-07-30 14:14:17 IST LOG: could not send data to client: Broken pipe
2015-07-30 14:14:17 IST WARNING: terminating connection because of crash
of another server process
2015-07-30 14:14:17 IST DETAIL: The postmaster has commanded this server
process to roll back the current transaction and exit, because another
server process exited abnormally and possibly corrupted shared memory.
2015-07-30 14:14:17 IST HINT: In a moment you should be able to reconnect
to the database and repeat your command.
2015-07-30 14:14:17 IST LOG: could not send data to client: Broken pipe

The whole recovery took more than 2.5 hours to complete to come back to a
state to accept connections back. This caused considerable disruption in
normal service.

Hardware support says after analysis that there is no hardware faults
related to network or IO (disks). Also could not figure out any network
errors like IP conflicts/IP changes etc as well.

I would like to troubleshoot this further to figure out what caused this
problem.
Any help is appreciated in this regard and precautions I can take to
understand the problems if occurs in future.

Regards,
James

Responses

Browse pgsql-novice by date

  From Date Subject
Next Message Tom Lane 2015-08-01 14:13:18 Re: Possible Corrputed shared memory
Previous Message Amit S. 2015-07-29 07:15:21 Re: Changing the value of data_directory