From: | "Boylan, Ross" <Ross(dot)Boylan(at)ucsf(dot)edu> |
---|---|
To: | "pgsql-admin(at)postgresql(dot)org" <pgsql-admin(at)postgresql(dot)org> |
Subject: | PostgreSQL 9.1 "database system was interrupted; last known up at" |
Date: | 2015-05-26 02:01:36 |
Message-ID: | F1F13E14A610474196571953929C02096D0E40@ex08.net.ucsf.edu |
Views: | Raw Message | Whole Thread | Download mbox | Resend email |
Thread: | |
Lists: | pgsql-admin |
I had to power cycle my system because it became unresponsive. Now PosgtreSQL will not start. I would like advice about how to proceed; I think pg_resetxlog is my next step. I have made a copy of the current database files.
<log file="postgresql-9.1-main.log">
2015-05-25 10:44:21 PDT LOG: database system was interrupted; last known up at 2015-05-22 09:22:25 PDT
2015-05-25 10:44:21 PDT LOG: incomplete startup packet
2015-05-25 10:44:21 PDT FATAL: could not open file "/etc/ssl/certs/ssl-cert-snakeoil.pem": Permission denied
2015-05-25 10:44:21 PDT LOG: startup process (PID 5180) exited with exit code 1
2015-05-25 10:44:21 PDT LOG: aborting startup due to startup process failure
</log>
I am running PostgreSQL 9.1 on Debian wheezy aka 7 aka oldstable.
Installed via the Debian package. I think I accepted the defaults, and have not changed the configuration since.
Linux 3.2.0, stock Debian kernel, amd64.
Connect via emacs sql-postgresql or psql. An init script controls startup.
When the system became unresponsive I was able to ssh in; the X process had gone crazy and could not be killed. Most key file systems had been remounted read-only, and many commands (includiing shutdown and telinit) produced errors, often I/O errors, when run. The last kern.log entries showed a process being killed. There was quite a lot of inode deletion and log replaying on restart. The log message above was from just after the restart.
I have no backups*, but could recreate the database in the worst case. I haven't done anything with the database in at least a week, I think, and so if I could get back the state as of 5/22 that would be fine.
Filesystem is ext3 on dm-crypt on LVM.
The permission error on the snakeoil cert is weird, since it is readable by all. I'm guessing it's a side effect of the earlier problems.
Thanks for any guidance.
Ross Boylan
(*) It would be a bit of mess even if I did, since I use bacula with postgres as the database. I think I was also dumping the database when I had backup going.
From | Date | Subject | |
---|---|---|---|
Next Message | Tom Lane | 2015-05-26 02:15:32 | Re: PostgreSQL 9.1 "database system was interrupted; last known up at" |
Previous Message | Bruce Momjian | 2015-05-25 13:08:56 | Re: Minor upgrades of postgre in aws rds? |