Re: [RFC] Incremental backup v2: add backup profile to base backup

On Fri, Oct 3, 2014 at 12:08 PM, Marco Nenciarini
<marco(dot)nenciarini(at)2ndquadrant(dot)it> wrote:
> Il 03/10/14 17:53, Heikki Linnakangas ha scritto:
>> If we're going to need a profile file - and I'm not convinced of that -
>> is there any reason to not always include it in the backup?
>
> The main reason is to have a centralized list of files that need to be
> present. Without a profile, you have to insert some sort of placeholder
> for kipped files.

Why do you need to do that? And where do you need to do that?

It seems to me that there are three interesting operations:

1. Take a full backup. Basically, we already have this. In the
backup label file, make sure to note the newest LSN guaranteed to be
present in the backup.

2. Take a differential backup. In the backup label file, note the LSN
of the fullback to which the differential backup is relative, and the
newest LSN guaranteed to be present in the differential backup. The
actual backup can consist of a series of 20-byte buffer tags, those
being the exact set of blocks newer than the base-backup's
latest-guaranteed-to-be-present LSN. Each buffer tag is followed by
an 8kB block of data. If a relfilenode is truncated or removed, you
need some way to indicate that in the backup; e.g. include a buffertag
with forknum = -(forknum + 1) and blocknum = the new number of blocks,
or InvalidBlockNumber if removed entirely.

3. Apply a differential backup to a full backup to create an updated
full backup. This is just a matter of scanning the full backup and
the differential backup and applying the changes in the differential
backup to the full backup.

You might want combinations of these, like something that does 2+3 as
a single operation, for efficiency, or a way to copy a full backup and
apply a differential backup to it as you go. But that's it, right?
What else do you need?

--
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company