From: | Ishan Joshi <Ishan(dot)Joshi(at)amdocs(dot)com> |
---|---|
To: | Amit jain <amit7(dot)jain(at)gmail(dot)com> |
Cc: | "pgsql-admin(at)lists(dot)postgresql(dot)org" <pgsql-admin(at)lists(dot)postgresql(dot)org> |
Subject: | RE: Postgres server crash and not starting |
Date: | 2020-03-04 11:31:53 |
Message-ID: | AM6PR0602MB339877431CD084A78AA83DB986E50@AM6PR0602MB3398.eurprd06.prod.outlook.com |
Views: | Raw Message | Whole Thread | Download mbox | Resend email |
Thread: | |
Lists: | pgsql-admin |
Hi Amit,
Thanks for your response.
Space is already there on server as for Data 1 TB space available an for Postgres directries pg_commit, pg_stat,pg_xact etc have 2+ GB space available.
You might be right on disk corruption, I have also have same suspect and checking the same.
Thanks & Regards,
Ishan Joshi
From: Amit jain <amit7(dot)jain(at)gmail(dot)com>
Sent: Wednesday, March 4, 2020 4:52 PM
To: Ishan Joshi <Ishan(dot)Joshi(at)amdocs(dot)com>
Cc: pgsql-admin(at)lists(dot)postgresql(dot)org
Subject: Re: Postgres server crash and not starting
Most likely input / output error occurs when disk corruption error exist, check the disk health and make sure it's having available space too.
On Wed, Mar 4, 2020, 4:48 PM Ishan Joshi <Ishan(dot)Joshi(at)amdocs(dot)com<mailto:Ishan(dot)Joshi(at)amdocs(dot)com>> wrote:
Hi Team,
I am using Postgresql server v11.4 on RHEL 7. This is my load test environment and I have reach database 1TB of size.
Last night the server went down with errors "No Space left on Device" though there is about 2GB space available on FS.
2020-03-03 17:11:37.202 IST pocdb schema1 109 2020-02-27 15:21:11 IST pid:27464 xid:0 ip:10.236.113.37 ERROR: could not access status of transaction 0
2020-03-03 17:11:37.202 IST pocdb schema1 110 2020-02-27 15:21:11 IST pid:27464 xid:0 ip:10.236.113.37 DETAIL: Could not write to file "pg_commit_ts/5198" at offset 90112: No space left on device.
2020-03-03 17:11:37.202 IST pocdb schema1 111 2020-02-27 15:21:11 IST pid:27464 xid:0 ip:10.236.113.37 STATEMENT: UPDATE TEST SET STATUS = $1, LAST_UPDATE_DATE = CURRENT_TIMESTAMP WHERE ID = $2 AND STATUS = $4
2020-03-03 17:11:37.203 IST pocdb schema1 197 2020-02-27 15:23:51 IST pid:30398 xid:0 ip:10.236.113.36 ERROR: could not access status of transaction 0
2020-03-03 17:11:37.203 IST pocdb schema1 198 2020-02-27 15:23:51 IST pi2020-03-03 17:14:29.937 IST 13 2020-02-27 15:15:14 IST pid:17705 xid:0 ip: LOG: database system is shut down
I have tried to restart the postgres services many times but it is now failing with below message
2020-03-04 12:31:19.469 IST 1 2020-03-04 12:31:19 IST pid:21703 xid:0 ip: LOG: database system shutdown was interrupted; last known up at 2020-03-04 11:49:08 IST
2020-03-04 12:31:19.704 IST 2 2020-03-04 12:31:19 IST pid:21703 xid:0 ip: LOG: database system was not properly shut down; automatic recovery in progress
2020-03-04 12:31:19.706 IST 3 2020-03-04 12:31:19 IST pid:21703 xid:0 ip: LOG: redo starts at 136E/9E042C78
2020-03-04 12:32:07.312 IST 4 2020-03-04 12:31:19 IST pid:21703 xid:0 ip: LOG: invalid record length at 1371/6CF69FD8: wanted 24, got 0
2020-03-04 12:32:07.312 IST 5 2020-03-04 12:31:19 IST pid:21703 xid:0 ip: LOG: redo done at 1371/6CF69F88
2020-03-04 12:32:07.312 IST 6 2020-03-04 12:31:19 IST pid:21703 xid:0 ip: LOG: last completed transaction was at log time 2020-03-03 17:13:36.683476+02
2020-03-04 12:32:07.319 IST 7 2020-03-04 12:31:19 IST pid:21703 xid:0 ip: LOG: checkpoint starting: end-of-recovery immediate
2020-03-04 12:32:07.964 IST 8 2020-03-04 12:31:19 IST pid:21703 xid:0 ip: PANIC: could not flush dirty data: Input/output error
2020-03-04 12:33:14.712 IST 7 2020-03-04 12:31:18 IST pid:21697 xid:0 ip: LOG: startup process (PID 21703) was terminated by signal 6: Aborted
2020-03-04 12:33:14.712 IST 8 2020-03-04 12:31:18 IST pid:21697 xid:0 ip: LOG: aborting startup due to startup process failure
2020-03-04 12:33:16.358 IST 9 2020-03-04 12:31:18 IST pid:21697 xid:0 ip: LOG: database system is shut down
Can you please suggest way to startup the postgres server.
Thanks & Regards,
Ishan Joshi
This email and the information contained herein is proprietary and confidential and subject to the Amdocs Email Terms of Service, which you may review at https://www.amdocs.com/about/email-terms-of-service<https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.amdocs.com%2Fabout%2Femail-terms-of-service&data=02%7C01%7CIshan.Joshi%40amdocs.com%7C7a262e0cda874f9f081308d7c02e84ce%7Cc8eca3ca127646d59d9da0f2a028920f%7C0%7C0%7C637189178344983255&sdata=OOrxmUM%2F2iQqdWA95YSqYX0TT%2FYBflIAWzhu9gjVbis%3D&reserved=0>
This email and the information contained herein is proprietary and confidential and subject to the Amdocs Email Terms of Service, which you may review at https://www.amdocs.com/about/email-terms-of-service <https://www.amdocs.com/about/email-terms-of-service>
From | Date | Subject | |
---|---|---|---|
Next Message | postggen2020 s | 2020-03-04 18:38:55 | Suggestion to reduce COPY command output to csv file |
Previous Message | Amit jain | 2020-03-04 11:22:06 | Re: Postgres server crash and not starting |