Re: help troubleshooting invalid page header error

Hi Chiru,

I am trying to pg_dump the database to have a snapshot of the current
state. I've turned on 'zero_damaged_pages' but pg_dump is still failing
with an "invalid page header" error - this time from what looks like a
sequence object that is auto-setting IDs on a table. Any advice on how to
remove this error?

Here is the full query that's failing:

SELECT sequence_name, start_value, last_value, increment_by, CASE WHEN
increment_by > 0 AND max_value = 9223372036854775807 THEN NULL WHEN
increment_by < 0 AND max_value = -1 THEN NULL ELSE max_value END AS
max_value, CASE WHEN increment_by > 0 AND min_value = 1 THEN NULL WHEN
increment_by < 0 AND min_value = -9223372036854775807 THEN NULL ELSE
min_value END AS min_value, cache_value, is_cycled, is_called from
unfinishedsubmissionstub_id_seq

On Fri, Dec 26, 2014 at 2:35 PM, chiru r <chirupg(at)gmail(dot)com> wrote:

> Hi Cory,
>
> After recovering table turn off *zero_damaged_pages *parameter.
>
>
> On Fri, Dec 26, 2014 at 9:13 PM, Cory Zue <czue(at)dimagi(dot)com> wrote:
>
>> Hi all,
>>
>> Thanks for the responses. Chiru, I'm looking into your suggestion.
>>
>> Sameer, here is the kernel version info:
>>
>> Linux dimagi 2.6.32-431.20.5.el6.x86_64 #1 SMP Wed Jul 16 05:26:53 EDT
>> 2014 x86_64 x86_64 x86_64 GNU/Linux
>>
>> Does that seem like it could be a problematic version?
>>
>> More generally - I'm still wondering whether I should chalk this failure
>> up to a transient/random issue, or whether I should be more worried about
>> the hardware on the machine. According to our diagnostic tools, disk and
>> memory are fine, but it's still not clear to me how it got into this state.
>> Any general bits of information regarding the potential causes of these
>> types of issues would be much appreciated.
>>
>> thanks,
>> Cory
>>
>>
>> On Fri, Dec 26, 2014 at 6:55 AM, Sameer Kumar <sameer(dot)kumar(at)ashnik(dot)com>
>> wrote:
>>
>>> On 23 Dec 2014 12:05, "Cory Zue" <czue(at)dimagi(dot)com> wrote:
>>> >
>>> > Hi all,
>>> >
>>> > Our postgres instance on one of our production machines has recently
>>> been returning errors of the form "DatabaseError: invalid page header in
>>> block 1 of relation base/16384/76623" from normal queries. I've been
>>> reading that these are often linked to hardware errors, but I would like to
>>> better understand what else it could be or how to determine that for sure.
>>> I've filled out the standard issue reporting template below. Any feedback
>>> or troubleshooting instructions would be much appreciated.
>>> >
>>> > ---
>>> > A description of what you are trying to achieve and what results you
>>> expect.:
>>> >
>>> > Intermittent queries are failing with the error "DatabaseError:
>>> invalid page header in block 1 of relation base/16384/76623"
>>> >
>>> > PostgreSQL version number you are running:
>>> >
>>> > PostgreSQL 8.4.13 on x86_64-redhat-linux-gnu, compiled by GCC gcc
>>> (GCC) 4.4.6 20120305 (Red Hat 4.4.6-4), 64-bit
>>> >
>>> > How you installed PostgreSQL:
>>> >
>>> > from standard package installer
>>> >
>>> > Changes made to the settings in the postgresql.conf file:
>>> >
>>> >
>>> > name | current_setting |
>>> source
>>> >
>>> ------------------------------+-----------------------------+----------------------
>>> > checkpoint_completion_target | 0.9 |
>>> configuration file
>>> > checkpoint_segments | 32 |
>>> configuration file
>>> > checkpoint_timeout | 15min |
>>> configuration file
>>> > DateStyle | ISO, MDY |
>>> configuration file
>>> > default_text_search_config | pg_catalog.english |
>>> configuration file
>>> > effective_cache_size | 1GB |
>>> configuration file
>>> > lc_messages | en_US.UTF-8 |
>>> configuration file
>>> > lc_monetary | en_US.UTF-8 |
>>> configuration file
>>> > lc_numeric | en_US.UTF-8 |
>>> configuration file
>>> > lc_time | en_US.UTF-8 |
>>> configuration file
>>> > log_checkpoints | on |
>>> configuration file
>>> > log_connections | off |
>>> configuration file
>>> > log_destination | csvlog |
>>> configuration file
>>> > log_directory | /opt/data/pgsql/data/pg_log |
>>> configuration file
>>> > log_disconnections | off |
>>> configuration file
>>> > log_duration | on |
>>> configuration file
>>> > log_filename | postgres-%Y-%m-%d_%H%M%S |
>>> configuration file
>>> > log_lock_waits | on |
>>> configuration file
>>> > log_min_duration_statement | 250ms |
>>> configuration file
>>> > log_rotation_age | 1d |
>>> configuration file
>>> > log_rotation_size | 1GB |
>>> configuration file
>>> > log_temp_files | 0 |
>>> configuration file
>>> > log_timezone | Asia/Kolkata | command
>>> line
>>> > log_truncate_on_rotation | on |
>>> configuration file
>>> > logging_collector | on |
>>> configuration file
>>> > maintenance_work_mem | 768MB |
>>> configuration file
>>> > max_connections | 500 |
>>> configuration file
>>> > max_stack_depth | 2MB |
>>> environment variable
>>> > port | 5432 | command
>>> line
>>> > shared_buffers | 4GB |
>>> configuration file
>>> > ssl | on |
>>> configuration file
>>> > TimeZone | Asia/Kolkata | command
>>> line
>>> > timezone_abbreviations | Default | command
>>> line
>>> > wal_buffers | 16MB |
>>> configuration file
>>> > work_mem | 48MB |
>>> configuration file
>>> >
>>> > It's also probably worth noting that postgres is installed on an
>>> encrypted volume which is mounted using ecryptfs.
>>> >
>>> > Operating system and version:
>>> >
>>> > RedHatEnterpriseServer, version 6.6
>>> >
>>> > What program you're using to connect to PostgreSQL:
>>> >
>>> > Python (django)
>>> >
>>> > Is there anything relevant or unusual in the PostgreSQL server logs?:
>>> >
>>> > I see lots of instances of this error (and similar). I'm not sure what
>>> else I should be looking for.
>>> >
>>> > What you were doing when the error happened / how to cause the error:
>>> >
>>> > I haven't explicitly tried to reproduce it, but it seems to
>>> consistently happen with certain queries. However, the system was rebooted
>>> shortly before the errors started occuring. The system was rebooted because
>>> another database (elasticsearch) was having problems on the same machine
>>> and the reboot was to attempt to resolve things.
>>> >
>>> > The EXACT TEXT of the error message you're getting, if there is one:
>>> >
>>> > DatabaseError: invalid page header in block 1 of relation
>>> base/16384/76623
>>> >
>>> > (block and relation numbers change)
>>> >
>>> > Unfortunately, I'm not completely familiar with the CPU and disk/RAID
>>> configurations used on the server. However it is storing to a (software)
>>> encrypted volume as mentioned above.
>>> >
>>> > Have you ever set fsync=off in the postgresql config file?
>>> > No
>>> > Have you had any unexpected power loss lately? Replaced a failed RAID
>>> disk? Had an operating system crash?
>>> > Not recently, though the system did reboot normally as described above.
>>> > Have you run a file system check? (chkdsk / fsck)
>>> > No.
>>> > Are there any error messages in the system logs?
>>> (unix/linux: dmesg, /var/log/syslog ;
>>> > I haven't seen anything obvious but I wasn't sure what to look for.
>>> >
>>>
>>> I guess you missed to provide the details and kernel version (rhel
>>> version and kernel level).
>>> This will give you kernel patch level-
>>>
>>> uname -a
>>>
>>> I had once faced this issue and I was on a buggy patch of Linux kernel.
>>> I just had to update to latest patch. That worked for me.
>>>
>>
>>
>