PostgreSQL 9.1 "database system was interrupted; last known up at"

From: "Boylan, Ross" <Ross(dot)Boylan(at)ucsf(dot)edu>
To: "pgsql-admin(at)postgresql(dot)org" <pgsql-admin(at)postgresql(dot)org>
Subject: PostgreSQL 9.1 "database system was interrupted; last known up at"
Date: 2015-05-26 02:01:36
Message-ID: F1F13E14A610474196571953929C02096D0E40@ex08.net.ucsf.edu
Views: Raw Message | Whole Thread | Download mbox | Resend email
Thread:
Lists: pgsql-admin

I had to power cycle my system because it became unresponsive. Now PosgtreSQL will not start. I would like advice about how to proceed; I think pg_resetxlog is my next step. I have made a copy of the current database files.

<log file="postgresql-9.1-main.log">
2015-05-25 10:44:21 PDT LOG: database system was interrupted; last known up at 2015-05-22 09:22:25 PDT
2015-05-25 10:44:21 PDT LOG: incomplete startup packet
2015-05-25 10:44:21 PDT FATAL: could not open file "/etc/ssl/certs/ssl-cert-snakeoil.pem": Permission denied
2015-05-25 10:44:21 PDT LOG: startup process (PID 5180) exited with exit code 1
2015-05-25 10:44:21 PDT LOG: aborting startup due to startup process failure
</log>

I am running PostgreSQL 9.1 on Debian wheezy aka 7 aka oldstable.
Installed via the Debian package. I think I accepted the defaults, and have not changed the configuration since.
Linux 3.2.0, stock Debian kernel, amd64.
Connect via emacs sql-postgresql or psql. An init script controls startup.

When the system became unresponsive I was able to ssh in; the X process had gone crazy and could not be killed. Most key file systems had been remounted read-only, and many commands (includiing shutdown and telinit) produced errors, often I/O errors, when run. The last kern.log entries showed a process being killed. There was quite a lot of inode deletion and log replaying on restart. The log message above was from just after the restart.

I have no backups*, but could recreate the database in the worst case. I haven't done anything with the database in at least a week, I think, and so if I could get back the state as of 5/22 that would be fine.

Filesystem is ext3 on dm-crypt on LVM.

The permission error on the snakeoil cert is weird, since it is readable by all. I'm guessing it's a side effect of the earlier problems.

Thanks for any guidance.

Ross Boylan

(*) It would be a bit of mess even if I did, since I use bacula with postgres as the database. I think I was also dumping the database when I had backup going.

Responses

Browse pgsql-admin by date

  From Date Subject
Next Message Tom Lane 2015-05-26 02:15:32 Re: PostgreSQL 9.1 "database system was interrupted; last known up at"
Previous Message Bruce Momjian 2015-05-25 13:08:56 Re: Minor upgrades of postgre in aws rds?