BUG #5185: postgres error "invalid page header in block xx of relation "pg_proc_proname_args_nsp_index"

The following bug has been logged online:

Bug reference: 5185
Logged by: Siwat Siwarborvorn
Email address: pg-support(at)xsidekick(dot)com
PostgreSQL version: 8.3.1
Operating system: Aix 5.3 64bit
Description: postgres error "invalid page header in block xx of
relation "pg_proc_proname_args_nsp_index"
Details:

Machine & OS Information
------------------
- AIX 5.3 64 bit

/home/admin >uname -a
AIX AYUD3 3 5 00CE5C1F4C00
/home/admin >oslevel -s
5300-09-03-0918

- Machine is IBM P570 with 16 CPU, Memory 16GB

- postgresql 8.3.1 compile with options

export CC="gcc -maix64"
export LDFLAGS='-Wl,-bbigtoc'
export OBJECT_MODE=64
export PATH=/usr/local/bin:$PATH
./configure --prefix=/usr/local/pgsql-8.3.1 --without-readline
--without-zlib \
--enable-thread-safety
make
make install

- We create database cluster on filesystem which is on EMC SAN.

- database cluster is on direcotry /pg

/home/admin >df /pg
Filesystem 512-blocks Free %Used Iused %Iused Mounted on
/dev/lvpgpro 524288000 134222744 75% 14265 1% /pg

- Currently we have about 700 connections run concurrently.

Scenario
--------
See "invalid page header" error in postgres log.

Error Example
-------
somchai:2009-11-11 16:35:29 THAIST:ERROR: invalid page header in block 49
of relation "pg_proc_proname_args_nsp_index"
somchai:2009-11-11 16:35:43 THAIST:ERROR: invalid page header in block 49
of relation "pg_proc_proname_args_nsp_index"

Bug Information
---------------

1) It happend on 2nd Nov, 3rd Nov and 11th Nov.

2) The error messages are similar. They show error on the same index,
pg_proc_proname_args_nsp_index.

3) On each day the messages show on the same block +1. But the block
numbers
are difference on each day they happened
For example,

On 2nd Nov, the error messages show on block # 13,14.
On 11th Nov, The error massages show on block # 48,49.

4) Only queries use the error block(s) on the index will failed.

5) We usually shutdown the machine on 6.00pm and boot on 6.30am
On the 1st and 3rd times, we see the error after 02.00pm. We quite sure
that
the index was ok and user could use it before we see the error. However on
2nd
time it happend right away when a user used the index.

6) Every time we fixe by reindex pg_proc_proname_args_nsp_index.

7) On 7th Nov, We ran pg_dump on the database and restored to a new
database cluster on a new file system to make sure that this time we won't
use the same disk area.

8) No error on AIX and EMC SAN

Postgresql config file ( Skip lines begin with #)
----------------------

listen_addresses = 'localhost, 172.1.1.7'
# (change requires restart)
port = 5432 # (change requires restart)
max_connections = 1000 # (change requires restart)

shared_buffers = 3GB # min 128kB or max_connections*16kB

max_fsm_pages = 500000 # min max_fsm_relations*16, 6 bytes each
# (change requires restart)
max_fsm_relations = 2000 # min 100, ~70 bytes each
# (change requires restart)
checkpoint_segments = 9 # in logfile segments, min 1, 16MB each

random_page_cost = 3.0 # same scale as above

logging_collector = on # Enable capturing of stderr and csvlog

log_directory = 'pg_log' # Directory where log files are
written
# Can be absolute or relative to
PGDATA
log_filename = 'postgresql-%Y-%m-%d_%H%M%S.log' # Log file name pattern.
# happen after that much log
# output. 0 to disable.
log_rotation_size = 0

log_line_prefix = '%u:%t:' # Special values:
# %u = user name
# e.g. '<%u%%%d> '
log_statement = 'mod' # none, ddl, mod, all

track_activities = on
track_counts = on

autovacuum = on # enable autovacuum subprocess?
# '' uses only default tablespace
datestyle = 'iso, dmy'
client_encoding = win874 # actually, defaults to database
# encoding

lc_messages = 'en_US' # locale for system error message
# strings
lc_monetary = 'en_US' # locale for monetary formatting
lc_numeric = 'en_US' # locale for number formatting
lc_time = 'en_US' # locale for time
formatting

default_text_search_config = 'pg_catalog.english'

default_with_oids = on