Re: Re: TODO list

On Thu, Apr 05, 2001 at 06:25:17PM -0400, Tom Lane wrote:
> "Mikheev, Vadim" <vmikheev(at)SECTORBASE(dot)COM> writes:
> >> If the reason that a block CRC isn't on the TODO list is that Vadim
> >> objects, maybe we should hear some reasons why he objects? Maybe
> >> the objections could be dealt with, and everyone satisfied.
>
> > Unordered disk writes are covered by backing up modified blocks
> > in log. It allows not only catch such writes, as would CRC do,
> > but *avoid* them.
>
> > So, for what CRC could be used? To catch disk damages?
> > Disk has its own CRC for this.
>
> Blocks that have recently been written, but failed to make it down to
> the disk platter intact, should be restorable from the WAL log. So we
> do not need a block-level CRC to guard against partial writes.

If a block is missing some sectors in the middle, how would you know
to reconstruct it from the WAL, without a block CRC telling you that
the block is corrupt?

> A block-level CRC might be useful to guard against long-term data
> lossage, but Vadim thinks that the disk's own CRCs ought to be
> sufficient for that (and I can't say I disagree).

The people who make the disks don't agree.

They publish the error rate they guarantee, and they meet it, more
or less. They publish a rate that is _just_ low enough to satisfy
noncritical requirements (on the correct assumption that they can't
satisfy critical requirements in any case) and high enough not to
interfere with benchmarks. They assume that if you need better
reliability you can and will provide it yourself, and rely on their
CRC only as a performance optimization.

At the raw sector level, they get (and correct) errors very frequently;
when they are not getting "enough" errors, they pack the bits more
densely until they do, and sell a higher-density drive.

> So the only real benefit of a block-level CRC would be to guard against
> bits dropped in transit from the disk surface to someplace else, ie,
> during read or during a "cp -r" type copy of the database to another
> location. That's not a totally negligible risk, but is it worth the
> overhead of updating and checking block CRCs? Seems dubious at best.

Vadim didn't want to re-open this discussion until after 7.1 is out
the door, but that "dubious at best" demands an answer. See the archive
posting:

http://www.postgresql.org/mhonarc/pgsql-hackers/2001-01/msg00473.html

...

Incidentally, is the page at

http://www.postgresql.org/mhonarc/pgsql-hackers/2001-01/

the best place to find old messages? It's never worked right for me.

Nathan Myers
ncm(at)zembu(dot)com