From: | Rushabh Lathia <rushabh(dot)lathia(at)gmail(dot)com> |
---|---|
To: | Robert Haas <robertmhaas(at)gmail(dot)com>, PostgreSQL Hackers <pgsql-hackers(at)postgresql(dot)org> |
Cc: | Jeevan Chalke <jeevan(dot)chalke(at)enterprisedb(dot)com>, vignesh C <vignesh21(at)gmail(dot)com>, David Steele <david(at)pgmasters(dot)net> |
Subject: | Re: backup manifests |
Date: | 2019-11-19 10:00:17 |
Message-ID: | CAGPqQf3DcmB6OtTpwqayH5bJgMnx2j-r4v2dFR6_=iFiqPq1Sw@mail.gmail.com |
Views: | Raw Message | Whole Thread | Download mbox | Resend email |
Thread: | |
Lists: | pgsql-hackers |
My colleague Suraj did testing and noticed the performance impact
with the checksums. On further testing, he found that specifically with
sha its more of performance impact.
Please find below statistics:
no of tables without checksum SHA256
checksum % performnce
overhead
with
SHA-256 md5 checksum % performnce
overhead with md5 CRC checksum % performnce
overhead with
CRC
10 (100 MB
in each table) real 0m10.957s
user 0m0.367s
sys 0m2.275s real 0m16.816s
user 0m0.210s
sys 0m2.067s 53% real 0m11.895s
user 0m0.174s
sys 0m1.725s 8% real 0m11.136s
user 0m0.365s
sys 0m2.298s 2%
20 (100 MB
in each table) real 0m20.610s
user 0m0.484s
sys 0m3.198s real 0m31.745s
user 0m0.569s
sys 0m4.089s
54% real 0m22.717s
user 0m0.638s
sys 0m4.026s 10% real 0m21.075s
user 0m0.538s
sys 0m3.417s 2%
50 (100 MB
in each table) real 0m49.143s
user 0m1.646s
sys 0m8.499s real 1m13.683s
user 0m1.305s
sys 0m10.541s 50% real 0m51.856s
user 0m0.932s
sys 0m7.702s 6% real 0m49.689s
user 0m1.028s
sys 0m6.921s 1%
100 (100 MB
in each table) real 1m34.308s
user 0m2.265s
sys 0m14.717s real 2m22.403s
user 0m2.613s
sys 0m20.776s 51% real 1m41.524s
user 0m2.158s
sys 0m15.949s
8% real 1m35.045s
user 0m2.061s
sys 0m16.308s 1%
100 (1 GB
in each table) real 17m18.336s
user 0m20.222s
sys 3m12.960s real 24m45.942s
user 0m26.911s
sys 3m33.501s 43% real 17m41.670s
user 0m26.506s
sys 3m18.402s 2% real 17m22.296s
user 0m26.811s
sys 3m56.653s
sometimes, this test
completes within the
same time as without
checksum. approx. 0.5%
Considering the above results, I modified the earlier Robert's patch and
added
"manifest_with_checksums" option to pg_basebackup. With a new patch.
by default, checksums will be disabled and will be only enabled when
"manifest_with_checksums" option is provided. Also re-based all patch set.
Regards,
--
Rushabh Lathia
www.EnterpriseDB.com
On Tue, Oct 1, 2019 at 5:43 PM Robert Haas <robertmhaas(at)gmail(dot)com> wrote:
> On Mon, Sep 30, 2019 at 5:31 AM Jeevan Chalke
> <jeevan(dot)chalke(at)enterprisedb(dot)com> wrote:
> > Entry for directory is not added in manifest. So it might be difficult
> > at client to get to know about the directories. Will it be good to add
> > an entry for each directory too? May be like:
> > Dir <dirname> <mtime>
>
> Well, what kind of corruption would this allow us to detect that we
> can't detect as things stand? I think the only case is an empty
> directory. If it's not empty, we'd have some entries for the files in
> that directory, and those files won't be able to exist unless the
> directory does. But, how would we end up backing up an empty
> directory, anyway?
>
> I don't really *mind* adding directories into the manifest, but I'm
> not sure how much it helps.
>
> --
> Robert Haas
> EnterpriseDB: http://www.enterprisedb.com
> The Enterprise PostgreSQL Company
>
>
>
--
Rushabh Lathia
Attachment | Content-Type | Size |
---|---|---|
0001-Reduce-code-duplication-and-eliminate-weird-macro-tr.patch | text/x-patch | 34.8 KB |
0004-Make-checksum-optional-in-pg_basebackup.patch | text/x-patch | 8.2 KB |
0003-Fix-warnings.patch | text/x-patch | 1.8 KB |
0002-POC-of-backup-manifest-with-file-names-sizes-timesta.patch | text/x-patch | 21.0 KB |
From | Date | Subject | |
---|---|---|---|
Next Message | Benjie Gillam | 2019-11-19 10:27:55 | Re: fix for BUG #3720: wrong results at using ltree |
Previous Message | Jinbao Chen | 2019-11-19 09:56:06 | Re: Planner chose a much slower plan in hashjoin, using a large table as the inner table. |