Re: block-level incremental backup

Greetings,

* Robert Haas (robertmhaas(at)gmail(dot)com) wrote:
> On Tue, Sep 17, 2019 at 12:09 PM Stephen Frost <sfrost(at)snowman(dot)net> wrote:
> > We need to be sure that we can detect if the WAL level has ever been set
> > to minimal between a full and an incremental and, if so, either refuse
> > to run the incremental, or promote it to a full, or make it a
> > checksum-based incremental instead of trusting the WAL stream.
>
> Sure. What about checksum collisions?

Certainly possible, of course, but a sha256 of each file is at least
somewhat better than, say, our page-level checksums. I do agree that
having the option to just say "promote it to a full", or "do a
byte-by-byte comparison against the prior backed up file" would be
useful for those who are concerned about sha256 collision probabilities.

Having a cross-check of "does this X% of files that we decided not to
back up due to whatever really still match what we think is in the
backup?" is definitely a valuable feature and one which I'd hope we get
to at some point.

> > Ok.. I can understand that but I don't get how these changes to
> > pg_basebackup will help facilitate that. If they don't and what you're
> > talking about here is independent, then great, that clarifies things,
> > but if you're saying that these changes to pg_basebackup are to help
> > with backing up directly into those Enterprise systems then I'm just
> > asking for some help in understanding how- what's the use-case here that
> > we're adding to pg_basebackup that makes it work with these Enterprise
> > systems?
> >
> > I'm not trying to be difficult here, I'm just trying to understand.
>
> Man, I feel like we're totally drifting off into the weeds here. I'm
> not arguing that these changes to pg_basebackup will help enterprise
> users except insofar as those users want incremental backup. All of
> this discussion started with this comment from you:
>
> "Having a system of keeping track of which backups are full and which
> are differential in an overall system also gives you the ability to do
> things like expiration in a sensible way, including handling WAL
> expiration."
>
> All I was doing was saying that for an enterprise user, the overall
> system might be something entirely outside of our control, like
> NetBackup or Tivoli. Therefore, whatever functionality we provide to
> do that kind of thing should be able to be used in such contexts. That
> hardly seems like a controversial proposition.

And all I was trying to understand was how what pg_basebackup does in
this context is really different from what can be done with pgbackrest,
since you brought it up:

"True, but I'm not sure that functionality belongs in core. It
certainly needs to be possible for out-of-core code to do this part of
the work if desired, because people want to integrate with enterprise
backup systems, and we can't come in and say, well, you back up
everything else using Netbackup or Tivoli, but for PostgreSQL you have
to use pg_backrest. I mean, maybe you can win that argument, but I
know I can't."

What it sounds like you're argueing here is that what pg_basebackup
"has" in it is that it specifically doesn't include any kind of
expiration management of any kind, and that's somehow helpful to people
who want to use Enterprise backup solutions. Maybe that's what you were
getting at, in which case, I'm sorry for misunderstanding and dragging
it out, and thanks for helping me understand.

> > How would that tool work, if it's to be able to work regardless of where
> > the WAL is actually stored..? Today, pg_archivecleanup just works
> > against a POSIX filesystem- are you thinking that the tool would have a
> > pluggable storage system, so that it could work with, say, a POSIX
> > filesystem, or a CIFS mount, or a s3-like system?
>
> Again, I was making a general statement about design goals -- "we
> should try to work nicely with enterprise backup products" -- not
> proposing a specific design for a specific thing. I don't think the
> idea of some pluggability in that area is a bad one, but it's not even
> slightly what this thread is about.

Well, I agree with you, as I said up-thread, that this seemed to be
going in a different and perhaps not entirely relevant direction.

> > Provided the WAL level is at the level that you need it to be that will
> > be true for things which are actually supported with PITR, replication
> > to standby servers, et al. I can see how it might come across as an
> > overreaction but this strikes me as a pretty glaring issue and I worry
> > that if it was overlooked until now that there'll be other more subtle
> > issues, and backups are just plain complicated to get right, just to
> > begin with already, something that I don't think people appreciate until
> > they've been dealing with them for quite a while.
>
> Permit me to be unpersuaded. If it was such a glaring issue, and if
> experience is the key to spotting such issues, then why didn't YOU
> spot it?

I'm not designing the feature..? Sure, I agreed earlier with the
general idea that we might be able to use WAL scanning and/or the LSN to
figure out if a page had changed, but the next step would have been, I
would have thought anyway, for someone to go do the analysis that has
only recently been started to look at the places when we write and the
cases where we write the WAL and actually build up confidence that this
approach isn't missing anything. Instead, we seem to have come a long
way in the development of this without having done that, and that does
shake my confidence in this effort.

> I'm not arguing that this stuff isn't hard. It is. Nor am I arguing
> that I didn't screw up. I did. But designs need to be accepted or
> rejected based on facts, not FUD. You've raised some good technical
> points and if you've got more concerns, I'd like to hear them, but I
> don't think arguing vaguely that a certain approach will probably run
> into trouble gets us anywhere.

This just gets back to what I was saying earlier. It seems like we're
presuming this is going to 'just work' because, say, replication works
great, or crash recovery works great, and those are based on WAL. I'm
still hopeful that we can do something based on WAL or LSN here, but it
needs a careful review of when we are, and when we aren't, writing out
WAL for basically everything we do, an effort that I'm glad to see might
be starting to happen, but a quick "oh, this is why in this one case
with this one thing, and we're all good now" doesn't instill confidence
in me, at least.

Thanks,

Stephen