Re: CRC algorithm (was Re: [REVIEW] Re: Compression of full-page-writes)

On Sat, Sep 13, 2014 at 1:33 AM, Heikki Linnakangas <hlinnakangas(at)vmware(dot)com>
wrote:
> On 09/12/2014 10:54 PM, Abhijit Menon-Sen wrote:
>> At 2014-09-12 22:38:01 +0300, hlinnakangas(at)vmware(dot)com wrote:
>>> We probably should consider switching to a faster CRC algorithm again,
>>> regardless of what we do with compression.
>>
>> As it happens, I'm already working on resurrecting a patch that Andres
>> posted in 2010 to switch to zlib's faster CRC implementation.
>
> As it happens, I also wrote an implementation of Slice-by-4 the other day
:-).
> Haven't gotten around to post it, but here it is.

Incase we are using the implementation for everything that uses
COMP_CRC32() macro, won't it give problem for older version
databases. I have created a database with Head code and then
tried to start server after applying this patch it gives below error:
FATAL: incorrect checksum in control file

In general, the idea sounds quite promising. To see how it performs
on small to medium size data, I have used attached test which is
written be you (with some additional tests) during performance test
of WAL reduction patch in 9.4.

Performance Data
------------------------------
Non-default settings

autovacuum = off
checkpoint_segments = 256
checkpoint_timeout = 20 min

HEAD -
testname | wal_generated | duration
-----------------------------------------+---------------+------------------
two short fields, no change | 583802008 | 11.6727559566498
two short fields, no change | 580888024 | 11.8558299541473
two short fields, no change | 580889680 | 11.5449349880219
two short fields, one changed | 620646400 | 11.6657111644745
two short fields, one changed | 620667904 | 11.6010649204254
two short fields, one changed | 622079320 | 11.6774570941925
two short fields, both changed | 620649656 | 12.0892491340637
two short fields, both changed | 620648360 | 12.1650269031525
two short fields, both changed | 620653952 | 12.2125108242035
one short and one long field, no change | 329018192 | 4.74178600311279
one short and one long field, no change | 329021664 | 4.71507883071899
one short and one long field, no change | 330326496 | 4.84932994842529
ten tiny fields, all changed | 701358488 | 14.236780166626
ten tiny fields, all changed | 701355328 | 14.0777900218964
ten tiny fields, all changed | 701358272 | 14.1000919342041
hundred tiny fields, all changed | 315656568 | 6.99316620826721
hundred tiny fields, all changed | 314875488 | 6.85715913772583
hundred tiny fields, all changed | 315263768 | 6.94613790512085
hundred tiny fields, half changed | 314878360 | 6.89090895652771
hundred tiny fields, half changed | 314877216 | 7.05924606323242
hundred tiny fields, half changed | 314881816 | 6.93445992469788
hundred tiny fields, half nulled | 236244136 | 6.43347096443176
hundred tiny fields, half nulled | 236248104 | 6.30539107322693
hundred tiny fields, half nulled | 236501040 | 6.33403086662292
9 short and 1 long, short changed | 262373616 | 4.24646091461182
9 short and 1 long, short changed | 262375136 | 4.49821400642395
9 short and 1 long, short changed | 262379840 | 4.38264393806458
(27 rows)

Patched -
testname | wal_generated | duration
-----------------------------------------+---------------+------------------
two short fields, no change | 580897400 | 10.6518769264221
two short fields, no change | 581779816 | 10.7118690013885
two short fields, no change | 581013224 | 10.8294110298157
two short fields, one changed | 620646264 | 10.8309078216553
two short fields, one changed | 620652872 | 10.8480410575867
two short fields, one changed | 620812376 | 10.9162290096283
two short fields, both changed | 620651792 | 10.9025599956512
two short fields, both changed | 620652304 | 10.7771129608154
two short fields, both changed | 620649960 | 11.0185468196869
one short and one long field, no change | 329022000 | 3.88278198242188
one short and one long field, no change | 329023656 | 4.01899003982544
one short and one long field, no change | 329022992 | 3.91587209701538
ten tiny fields, all changed | 701353296 | 12.7748699188232
ten tiny fields, all changed | 701354848 | 12.761589050293
ten tiny fields, all changed | 701356520 | 12.6703131198883
hundred tiny fields, all changed | 314879424 | 6.25606894493103
hundred tiny fields, all changed | 314878416 | 6.32905578613281
hundred tiny fields, all changed | 314878464 | 6.28877377510071
hundred tiny fields, half changed | 314874808 | 6.25019288063049
hundred tiny fields, half changed | 314881296 | 6.41510701179504
hundred tiny fields, half changed | 314881320 | 6.42809700965881
hundred tiny fields, half nulled | 236248928 | 5.9281849861145
hundred tiny fields, half nulled | 236251768 | 5.91391110420227
hundred tiny fields, half nulled | 236247288 | 5.94086098670959
9 short and 1 long, short changed | 262374536 | 3.77700018882751
9 short and 1 long, short changed | 262377504 | 3.81636500358582
9 short and 1 long, short changed | 262378880 | 3.84033012390137
(27 rows)

The patched version gives better results in all cases
(in range of 10~15%), though this is not the perfect test, however
it gives fair idea that the patch is quite promising. I think to test
the benefit from crc calculation for full page, we can have some
checkpoint during each test (may be after insert). Let me know
what other kind of tests do you think are required to see the
gain/loss from this patch.

I think the main difference in this patch and what Andres has
developed sometime back was code for manually unrolled loop
doing 32bytes at once, so once Andres or Abhijit will post an
updated version, we can do some performance tests to see
if there is any additional gain.

With Regards,
Amit Kapila.
EnterpriseDB: http://www.enterprisedb.com