Re: PANIC: corrupted item pointer

Hi,

thanks so much for answering. I found a "segmentation fault" in my logs
so please check below:

> On Tue, 2012-03-27 at 11:47 +0200, Janning Vygen wrote:
>>
>> I am running postgresql-9.1 from debian backport package fsync=on
>> full_page_writes=off
>
> That may be unsafe (and usually is) depending on your I/O system and
> filesystem. However, because you didn't have any power failures, I
> don't think this is the cause of the problem.

I think i should switch to full_page_writes=on. But as my harddisk are
rather cheap, so I used to tune it to get maximum performance.

> These WARNINGs below could also be caused by a power failure. Can
> you verify that no power failure occurred? E.g. check uptime, and
> maybe look at a few logfiles?

The PANIC occurred first on March, 19. My servers uptime ist 56 days, so
about 4th of February. There was no power failure since i started to use
this machine. This machine is in use since March, 7. I checked it twice:
Now power failure.

But i found more strange things, so let me show you a summary (some
things were shortened for readability)

1. Segmentation fault
Mar 13 19:01 LOG: server process (PID 32464) was terminated by signal
11: Segmentation fault
Mar 13 19:01 FATAL: the database system is in recovery mode
Mar 13 19:01 LOG: unexpected pageaddr 22/8D402000 in log file 35,
segment 208, offset 4202496
Mar 13 19:01 LOG: redo done at 23/D0401F78
Mar 13 19:01 LOG: last completed transaction was at log time 2012-03-13
19:01:58.667779+01
Mar 13 19:01 LOG: checkpoint starting: end-of-recovery immediate

2. PANICS
Mar 19 22:14 PANIC: corrupted item pointer: offset = 21248, size = 16
Mar 20 23:38 PANIC: corrupted item pointer: offset = 21248, size = 16
Mar 21 23:30 PANIC: corrupted item pointer: offset = 21248, size = 16
Mar 23 02:10 PANIC: corrupted item pointer: offset = 21248, size = 16
Mar 24 06:12 PANIC: corrupted item pointer: offset = 21248, size = 16
Mar 25 01:28 PANIC: corrupted item pointer: offset = 21248, size = 16
Mar 26 22:16 PANIC: corrupted item pointer: offset = 21248, size = 16
Mar 27 09:17 PANIC: corrupted item pointer: offset = 21248, size = 16
Mar 27 09:21 PANIC: corrupted item pointer: offset = 21248, size = 16
Mar 27 09:36 PANIC: corrupted item pointer: offset = 21248, size = 16
Mar 27 09:48 PANIC: corrupted item pointer: offset = 21248, size = 16
Mar 27 10:01 PANIC: corrupted item pointer: offset = 21248, size = 16

What I additionally see, that my table rankingentry was not autovacuumed
anymore after the first PANIC on March,19. But it was still autovacuumed
after segmentation fault without error.

3.
Then I rebuilt all index on this table, dropped old indexes, and did run
vacuum on this table:

WARNING: relation "rankingentry" page 424147 is uninitialized --- fixing
WARNING: relation "rankingentry" page 424154 is uninitialized --- fixing
WARNING: relation "rankingentry" page 424155 is uninitialized --- fixing
WARNING: relation "rankingentry" page 424166 is uninitialized --- fixing
WARNING: relation "rankingentry" page 424167 is uninitialized --- fixing
WARNING: relation "rankingentry" page 424180 is uninitialized --- fixing

After this everything is running just fine. No more problems, just headache.

> Did you get the PANIC and WARNINGs on the primary or the replica? It
> might be worth doing some comparisons between the two systems.

It only happend on my primary server. My backup server has no suspicious
log entries.

It is pretty obvious to me the segmentation fault is the main reason for
getting the PANIC afterwards. What can cause a segmentation fault? Is
there anything to analyse further?

kind regards
Janning

--
Kicktipp GmbH

Venloer Straße 8, 40477 Düsseldorf
Sitz der Gesellschaft: Düsseldorf
Geschäftsführung: Janning Vygen
Handelsregister Düsseldorf: HRB 55639

http://www.kicktipp.de/