Re: 8.3.5 broken after power fail SOLVED

Naomi Walker wrote:
> Other than disaster tests, how would I know if I have an system that
> lies about fsync?

Well, the linux kernel tries to detect it on bootup and
will give messages like this:
%dmesg | grep 'disabling barriers'
JBD: barrier-based sync failed on md1 - disabling barriers
JBD: barrier-based sync failed on hda3 - disabling barriers
when it detects certain types of unreliable fsync's. The command
%hdparm -I /dev/hdf | grep FLUSH_CACHE_EXT
will give you clues if a hard drive itself even can support
a non-lying fsync when it's internal cache is enabled.

Sadly some filesystems (ext3) lie even above and beyond what
Linux does - by only using the write barriers correctly
when the inode itself is modified; not when the data is modified.
A test program here:
http://archives.postgresql.org/pgsql-performance/2008-08/msg00159.php
can detect those cases where the kernel & drive don't lie
about fsync but ext3 lies in spite of them; with more background
info here:
http://article.gmane.org/gmane.linux.file-systems/21373
http://thread.gmane.org/gmane.linux.kernel/646040

Elsewhere in the archives you can find programs that measure
how fast fsyncs happen - but on your hardware, and you can
try to see if those numbers approximately match how fast your
disks spin. But then you still need to make sure the test
program used the same methods for syncing the drive that your
postgres configuration files are choosing.

I wonder if the only really safe way is to run a very
write intensive database script and pull and kill your
system in a number of ways, including yanking power to
the system; to disk arrays, etc and see if your database died.

>
>
>>
>> We preach this again and again. PostgreSQL can only survive a power
>> outage type failure ONLY if the hardware / OS / filesystem don't lie
>> about fsync. If they do, all bets are off, and this kind of failure
>> means you should really failover to another machine or restore a
>> backup.
>>
>> It's why you have to do possibly destructive tests to see if your
>> server stands at least some chance of surviving this kind of failure,
>> log shipping for recovery, and / or replication of another form (slony
>> etc...) to have a reliable server.
>>
>> The recommendations for recovery of data are just that, recovery
>> oriented. They can't fix a broken database at that point. You need
>> to take it offline after this kind of failure if you can't trust your
>> hardware.
>>
>> Usually when it finds something wrong it just won't start up.
>>
>>
>
>