Re: Block-level CRC checks

Martijn van Oosterhout wrote:
> On Fri, Nov 14, 2008 at 10:51:57AM -0500, Tom Lane wrote:
>> In fact, if the patch were to break torn-page handling, it would be
>> 100% likely to be a net *decrease* in system reliability. It would add
>> detection of a situation that is not supposed to happen (ie, storage
>> system fails to return the same data it stored) at the cost of breaking
>> one's database when the storage system acts as it's expected and
>> documented to in a routine power-loss situation.
>
> Ok, I see it's a problem because the hint changes are not WAL logged,
> so torn pages are expected to work in normal operation. But simply
> skipping the hint bits during checksumming is a terrible solution,
> since then any errors in those bits will go undetected. To not be able
> to say in the documentation that you'll detect 100% of single-bit
> errors is pretty darn terrible, since that's kind of the goal of the
> exercise.

Agreed, trying to explain that in the documentation would look like
making excuses.

The requirement that all hint bit changes are WAL-logged seems like a
pretty big change. I don't like doing that, just for CRCing.

There has been discussion before about not writing out pages to disk
that only have hint-bit updates on them. That means that the next time
the page is read, the reader needs to do the clog lookups and set the
hint bits again. It's a tradeoff, making the first SELECT after
modifying a page cheaper, I/O-wise, at the cost of making all subsequent
SELECTs that need to read the page from disk or kernel cache more
expensive, CPU-wise.

I'm not sure if I like that idea or not, but it would also solve the CRC
problem with torn pages. FWIW, it would also solve the problem suggested
with IBM DTLA disks and others that might zero-out a sector in case of
an interrupted write. I'm not totally convinced that's a problem, as
there's apparently other software that make the same assumption as we
do, and we haven't heard of any torn-page corruption in real life, but
still.

If we made the behavior configurable, that would be pretty hard to
explain in the docs. We'd have three options with dependencies

- CRC on/off
- write pages with only hint bit changes on/off
- full_page_writes on/off

If disable full_page_writes, you're vulnerable to torn pages. If you
enable it, you're not. Except if you also turn CRC on. Except if you
also turn "write pages with only hint bit changes" off.

> Unfortunatly, there's not a lot of easy solutions here. You could do
> two checksums, one with and one without hint bits. The overall checksum
> tells you if there's a problem. If it doesn't match the second checksum
> will tell you if it's the hint bits or not (torn page problem). If it's
> the hint bits you can reset them all and continue. The checksums need
> not be of equal strength.

Hmm, that would work I guess.

> The extreme case is an ECC where you explicitly can set it so you can
> alter N bits before you need to recalculate the checksum.
> Computationally though, that sucks.

Yep. Also, in case of a torn page, you're very likely going to have
several hint bits from the old image and several from the new image. An
error-correcting code would need to be unfeasibly long to cope with that.

--
Heikki Linnakangas
EnterpriseDB http://www.enterprisedb.com