Re: design for parallel backup

Thanks for your thoughts.

On Mon, Apr 20, 2020 at 4:02 PM Peter Eisentraut
<peter(dot)eisentraut(at)2ndquadrant(dot)com> wrote:
> That would clearly be a good goal. Non-parallel backup should ideally
> be parallel backup with one worker.

Right.

> But it doesn't follow that the proposed design is wrong. It might just
> be that the design of the existing backup should change.
>
> I think making the wire format so heavily tied to the tar format is
> dubious. There is nothing particularly fabulous about the tar format.
> If the server just sends a bunch of files with metadata for each file,
> the client can assemble them in any way they want: unpacked, packed in
> several tarball like now, packed all in one tarball, packed in a zip
> file, sent to S3, etc.

Yeah, that's true, and I agree that there's something a little
unsatisfying and dubious about the current approach. However, I am not
sure that there is sufficient reason to change it to something else,
either. After all, what purpose would such a change serve? The client
can already do any of the things you mention here, provided that it
can interpret the data sent by the server, and pg_basebackup already
has code to do exactly this. Right now, we have pretty good
pg_basebackup compatibility across server versions, and if we change
the format, then we won't, unless we make both the client and the
server understand both formats. I'm not completely averse to such a
change if it has sufficient benefits to make it worthwhile, but it's
not clear to me that it does.

> Another thing I would like to see sometime is this: Pull a minimal
> basebackup, start recovery and possibly hot standby before you have
> received all the files. When you need to access a file that's not there
> yet, request that as a priority from the server. If you nudge the file
> order a little with perhaps prewarm-like data, you could get a mostly
> functional standby without having to wait for the full basebackup to
> finish. Pull a file on request is a requirement for this.

True, but that can always be implemented as a separate feature. I
won't be sad if that feature happens to fall out of work in this area,
but I don't think the possibility that we'll some day have such
advanced wizardry should bias the design of this feature very much.
One pretty major problem with this is that you can't open for
connections until you've reached a consistent state, and you can't say
that you're in a consistent state until you've replayed all the WAL
generated during the backup, and you can't say that you're at the end
of the backup until you've copied all the files. So, without some
clever idea, this would only allow you to begin replay sooner; it
would not allow you to accept connections sooner. I suspect that makes
it significantly less appealing.

> > So, my new idea for parallel backup is that the server will return
> > tarballs, but just more of them. Right now, you get base.tar and
> > ${tablespace_oid}.tar for each tablespace. I propose that if you do a
> > parallel backup, you should get base-${N}.tar and
> > ${tablespace_oid}-${N}.tar for some or all values of N between 1 and
> > the number of workers, with the server deciding which files ought to
> > go in which tarballs.
>
> I understand the other side of this: Why not compress or encrypt the
> backup already on the server side? Makes sense. But this way seems
> weird and complicated. If I want a backup, I want one file, not an
> unpredictable set of files. How do I even know I have them all? Do we
> need a meta-manifest?

Yes, that's a problem, but...

> A format such as ZIP would offer more flexibility, I think. You can
> build a single target file incrementally, you can compress or encrypt
> each member file separately, thus allowing some compression etc. on the
> server. I'm not saying it's perfect for this, but some more thinking
> about the archive formats would potentially give some possibilities.

...I don't think this really solves anything. I expect you would have
to write the file more or less sequentially, and I think that Amdahl's
law will not be kind to us.

> All things considered, we'll probably want more options and more ways of
> doing things.

Yes. That's why I'm trying to figure out how to create a flexible framework.

Thanks,

--
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company