Re: backup manifests

Hi,

On 2020-03-26 15:37:11 -0400, Stephen Frost wrote:
> The argument is that adding checksums takes more time. I can understand
> that argument, though I don't really agree with it. Certainly a few
> percent really shouldn't be that big of an issue, and in many cases even
> a sha256 hash isn't going to have that dramatic of an impact on the
> actual overall time.

I don't understand how you can come to that conclusion? It doesn't take
very long to measure openssl's sha256 performance (which is pretty well
optimized). Note that we do use openssl's sha256, when compiled with
openssl support.

On my workstation, with a pretty new (but not fastest single core perf
model) intel Xeon Gold 5215, I get:

$ openssl speed sha256
...
type 16 bytes 64 bytes 256 bytes 1024 bytes 8192 bytes 16384 bytes
sha256 76711.75k 172036.78k 321566.89k 399008.09k 431423.49k 433689.94k

IOW, ~430MB/s.

On my laptop, with pretty fast cores:
type 16 bytes 64 bytes 256 bytes 1024 bytes 8192 bytes 16384 bytes
sha256 97054.91k 217188.63k 394864.13k 493441.02k 532100.44k 533441.19k

IOW, 530MB/s

530 MB/s is well within the realm of medium sized VMs.

And, as mentioned before. even if you do only half of that, you're still
going to be spending roughly half of the CPU time of sending a base
backup.

What makes you think that a few hundred MB/s is out of reach for a large
fraction of PG installations that actually keep backups?

Greetings,

Andres Freund