Re: Enabling Checksums

On Wed, Apr 10, 2013 at 12:25 PM, Simon Riggs <simon(at)2ndquadrant(dot)com> wrote:
> On 10 April 2013 09:01, Ants Aasma <ants(at)cybertec(dot)at> wrote:
>>
>>
>> > Using SIMD for WAL is not a requirement at all; I just thought it might
>> > be a nice benefit for non-checksum-enabled users in some later release.
>>
>> I think we should first deal with using it for page checksums and if
>> future versions want to reuse some of the code for WAL checksums then
>> we can rearrange the code.
>
>
> We have essentially the same problem in both cases: we want to calculate a
> checksum of BLCKSZ chunks of data, plus some smaller header data. We
> currently use the same code for both cases and it makes sense to do the same
> thing with any new code. This is also the *same* issue: when we make a new
> hint we need to issue a full page write in WAL, so we are calculating
> checksums in two new places: XLOG_HINT records and data blocks.
>
> Few technical points:
>
> * We're taking a copy of the buffer, so Jeff's zero trick works safely, I
> think.
> * We can use a different algorithm for big and small blocks, we just need a
> way to show we've done that, for example setting the high order bit of the
> checksum.
> * We might even be able to calculate CRC32 checksum for normal WAL records,
> and use Ants' checksum for full page writes (only). So checking WAL checksum
> would then be to confirm header passes CRC32 and then re-check the Ants
> checksum of each backup block.

If we ensure that the checksum on the page is correct when we do a
full page write then we could only include the checksum field in the
WAL CRC. When reading WAL we would first check that the CRC is correct
and then verify the the page checksum.

> This work needs to happen now, since once the checksum algorithm is set we
> won't easily be able to change it.

The page checksum algorithm needs to be decided now, but WAL CRCs and
full page writes can be changed in 9.4 and don't need to be perfect on
the first try.

Regards,
Ants Aasma
--
Cybertec Schönig & Schönig GmbH
Gröhrmühlgasse 26
A-2700 Wiener Neustadt
Web: http://www.postgresql-support.de