Re: Cost of XLogInsert CRC calculations

> -----Original Message-----
> From: Mark Cave-Ayland [mailto:m(dot)cave-ayland(at)webbased(dot)co(dot)uk]
> Sent: 07 March 2005 11:04
> To: 'tgl(at)sss(dot)pgh(dot)pa(dot)us'
> Cc: 'pgsql-hackers(at)postgreSQL(dot)org'
> Subject: Re: Cost of XLogInsert CRC calculations

(cut)

> > I suppose that the bulk of the CPU cycles being attributed to
> > XLogInsert are going into > the inlined CRC calculations. Maybe we
> > need to think twice about the cost/benefit ratio > of using 64-bit
> > CRCs to protect xlog records that are often only a few dozen bytes.
>
> Wow, a 64-bit CRC does seem excessive, especially when going
> back to Zmodem days where a 50-100k file seemed to be easily
> protected by a 32-bit CRC. I'm sure there are some error
> rates somewhere dependent upon the polynomial and the types
> of error detected.... Try the following link towards the
> bottom: http://www.ee.unb.ca/tervo/ee4253/crc.htm for some
> theory on detection rates vs. CRC size.

Hi Tom/Simon,

I was just researching some articles on compression (zlib) and I saw mention
of the Adler-32 algorithm which is supposed to be slightly less accurate
than an equivalent CRC calculation but significantly faster to compute. I
haven't located a good paper comparing the error rates of the two different
checksums, however extending the Adler-32 algorithm up to 64 bits may be a
way of increasing the speed at the expense of a slight loss in the accuracy
of error detection if we decided that we want to stay at 64 bits as opposed
to 32.

The following seems to indicate that Adler-32 is at least twice as fast as
optimised CRC32: http://www.winimage.com/misc/readfile_test.htm.

Kind regards,

Mark.

------------------------
WebBased Ltd
17 Research Way
Plymouth
PL6 8BT

T: +44 (0)1752 791021
F: +44 (0)1752 791023
W: http://www.webbased.co.uk