pgsql 8.3.3 "Invalid page header" leads to "FATAL: segment too big"

From: Achilleas Mantzios <achill(at)matrix(dot)gatewaynet(dot)com>
To: "pgsql-admin(at)postgresql(dot)org" <pgsql-admin(at)postgresql(dot)org>
Subject: pgsql 8.3.3 "Invalid page header" leads to "FATAL: segment too big"
Date: 2010-04-22 12:16:08
Message-ID: 201004221516.08729.achill@matrix.gatewaynet.com
Views: Raw Message | Whole Thread | Download mbox | Resend email
Thread:
Lists: pgsql-admin

Hello,
i have this serious problem in one of our remote vessels. (comm is done by minicom to the remote satelite modem)
I think that this server was under some sort of constant resets or hardware failures.
Initially,i had this problem:
ERROR: invalid page header in block 672720 of relation "pg_toast_125716009"

This toast table corresponds to a table named "mail_message",
Table "public.mail_message"
Column | Type | Modifiers
-----------+-------------------+-----------------------------------------------------------
msgno | mail_msgno_domain | not null default nextval('mail_msgno_sequence'::regclass)
msgsource | bytea |
Indexes:
"mail_message_key" PRIMARY KEY, btree (msgno)

(obviously the TOAST table serves the msgsource varlena) the contents of which is not of vital importance.
I tried, REINDEXING, with no success, and after that, i tried resetting the said block on disk as per this
suggestion by Tom here: http://old.nabble.com/invalid-page-header-td11981154.html

i found the oid of the table:
SELECT tableoid,1 from pg_toast.pg_toast_125716009 order by chunk_id LIMIT 1
tableoid | ?column?
-----------+----------
125716013 | 1

(and just to verify)
SELECT relname from pg_class where oid=125716013;
relname
--------------------
pg_toast_125716009

Then i did: (as i said i do not need the contents of msgsource - yet the contents of msgno are vital)
dd if=/dev/zero of=/usr/local/var/lib/pgsql/data/base/125714957/125716013 seek=672720 bs=8192 count=1

However, after that, unfortunately i get constant postgresql server restarts with:
FATAL: segment too big
server closed the connection unexpectedly
This probably means the server terminated abnormally
before or while processing the request.
The connection to the server was lost. Attempting reset: Succeeded.

Is there anything i can do to savage the situation?

(one of) the hard part here is that i dont have neither physical nor network access to the server
(only ultra expensive unreliable satellite comms)

Thanks for any hints...

--
Achilleas Mantzios

Responses

Browse pgsql-admin by date

  From Date Subject
Next Message Cédric Villemain 2010-04-22 13:53:05 Re: pgsql 8.3.3 "Invalid page header" leads to "FATAL: segment too big"
Previous Message Scott Marlowe 2010-04-22 06:45:07 Re: ALTER TABLE <name> ADD COLUMN Hangs