Re: pg_combinebackup does not detect missing files

On Sun, Apr 21, 2024 at 8:47 PM David Steele <david(at)pgmasters(dot)net> wrote:
> > I figured that wouldn't be particularly meaningful, and
> > that's pretty much the only kind of validation that's even
> > theoretically possible without a bunch of extra overhead, since we
> > compute checksums on entire files rather than, say, individual blocks.
> > And you could really only do it for the final backup in the chain,
> > because you should end up accessing all of those files, but the same
> > is not true for the predecessor backups. So it's a very weak form of
> > verification.
>
> I don't think it is weak if you can verify that the output is exactly as
> expected, i.e. all files are present and have the correct contents.

I don't understand what you mean here. I thought we were in agreement
that verifying contents would cost a lot more. The verification that
we can actually do without much cost can only check for missing files
in the most recent backup, which is quite weak. pg_verifybackup is
available if you want more comprehensive verification and you're
willing to pay the cost of it.

> I tested the patch and it works, but there is some noise from WAL files
> since they are not in the manifest:
>
> $ pg_combinebackup test/backup/full test/backup/incr1 -o test/backup/combine
> pg_combinebackup: warning: "pg_wal/000000010000000000000008" is present
> on disk but not in the manifest
> pg_combinebackup: error: "base/1/3596" is present in the manifest but
> not on disk

Oops, OK, that can be fixed.

> I think it is a worthwhile change and we are still a month away from
> beta1. We'll see if anyone disagrees.

I don't plan to press forward with this in this release unless we get
a couple of +1s from disinterested parties. We're now two weeks after
feature freeze and this is design behavior, not a bug. Perhaps the
design should have been otherwise, but two weeks after feature freeze
is not the time to debate that.

--
Robert Haas
EDB: http://www.enterprisedb.com