Ouch!

Running 8.1.3 on FreeBSD 6.0 server and got the following (I'm most
interested in the "PANIC" entry):

2007-02-06 17:15:11 CST LOG: archived transaction log file
"00000001000004A300000025"
2007-02-06 17:15:27 CST LOG: archived transaction log file
"00000001000004A300000026"
2007-02-06 17:15:39 CST LOG: archived transaction log file
"00000001000004A300000027"
2007-02-06 17:16:16 CST LOG: archived transaction log file
"00000001000004A300000028"
2007-02-06 17:16:31 CST LOG: archived transaction log file
"00000001000004A300000029"
2007-02-06 17:16:36 CST LOG: statistics buffer is full
2007-02-06 17:16:36 CST LOG: statistics buffer is full
2007-02-06 17:16:36 CST LOG: statistics buffer is full
2007-02-06 17:18:10 CST LOG: archived transaction log file
"00000001000004A30000002A"
2007-02-06 17:19:40 CST LOG: archived transaction log file
"00000001000004A30000002B"
2007-02-06 17:20:18 CST LOG: archived transaction log file
"00000001000004A30000002C"
2007-02-06 17:21:12 CST LOG: archived transaction log file
"00000001000004A30000002D"
2007-02-06 17:21:34 CST LOG: archived transaction log file
"00000001000004A30000002E"
2007-02-06 17:21:55 CST LOG: archived transaction log file
"00000001000004A30000002F"
2007-02-06 17:22:03 CST LOG: archived transaction log file
"00000001000004A300000030"
2007-02-06 17:22:22 CST PANIC: right sibling's left-link doesn't match
2007-02-06 17:22:22 CST STATEMENT: update winddir set value = 20, qc =
0, updated = '2007-02-06 17:18:22', revised = '2007-02-06 23:17:00',
actual = '2007-02-06 22:55:0
0', off_set = 300, feed = 'NP', feed_history = 'NPiNPu', qcd =
'1900-01-01 00:00:00' where station_id = 'KHEZ' and date = '2007-02-06
23:00:00' and value != 20 and ( ( o
ff_set > 300 ) or ( value = -9999 ) or ( ( actual = '2007-02-06
22:55:00' ) and ( revised > '2007-02-06 23:17:00' ) ) )
2007-02-06 17:22:24 CST LOG: server process (PID 58978) was terminated
by signal 6
2007-02-06 17:22:24 CST LOG: terminating any other active server processes
2007-02-06 17:22:24 CST WARNING: terminating connection because of
crash of another server process
2007-02-06 17:22:24 CST DETAIL: The postmaster has commanded this
server process to roll back the current transaction and exit, because
another server process exited ab
normally and possibly corrupted shared memory.
2007-02-06 17:22:24 CST HINT: In a moment you should be able to
reconnect to the database and repeat your command.
2007-02-06 17:22:24 CST WARNING: terminating connection because of
crash of another server process
2007-02-06 17:22:24 CST DETAIL: The postmaster has commanded this
server process to roll back the current transaction and exit, because
another server process exited ab
normally and possibly corrupted shared memory.
2007-02-06 17:22:24 CST HINT: In a moment you should be able to
reconnect to the database and repeat your command.

The server restarted successfully. Good. The server panicked again the
next time the same table was accessed. Boo. Of course this happened at
night - over and over again. Especial problematic during our PITR
archive cycle during which an archived WAL required by recovery process
was shuffled around preventing a successful restart.

I sorted out my archived WAL issues, brought the server up, and
reindexed the offending table. Life has been good ever since.

It looks like the server behaved admirably. However, I'm wondering if
there is any more light that can be shed on this. Archived inquiries
indicate a) there may be hardware issues causing this problem and b) a
potential software problem related to this condistion may have been
addressed in 8.2.n.

Any advice?

--
--------------------------------------------------------------------------------
Andrew Rost
National Operational Hydrologic Remote Sensing Center (NOHRSC)
National Weather Service, NOAA
1735 Lake Dr. West, Chanhassen, MN 55317-8582
Voice: (952)361-6610 x 234
Fax: (952)361-6634
andy(dot)rost(at)noaa(dot)gov
http://www.nohrsc.noaa.gov
--------------------------------------------------------------------------------