Re: [RFC] Incremental backup v2: add backup profile to base backup

On 10/03/2014 06:31 PM, Marco Nenciarini wrote:
> Hi Hackers,
>
> I've updated the wiki page
> https://wiki.postgresql.org/wiki/Incremental_backup following the result
> of discussion on hackers.
>
> Compared to first version, we switched from a timestamp+checksum based
> approach to one based on LSN.
>
> This patch adds an option to pg_basebackup and to replication protocol
> BASE_BACKUP command to generate a backup_profile file. It is almost
> useless by itself, but it is the foundation on which we will build the
> file based incremental backup (and hopefully a block based incremental
> backup after it).

I'd suggest jumping straight to block-based incremental backup. It's not
significantly more complicated to implement, and if you implement both
separately, then we'll have to support both forever. If you really need
to, you can implement file-level diff as a special case, where the
server sends all blocks in the file, if any of them have an LSN > the
cutoff point. But I'm not sure if there's point in that, once you have
block-level support.

If we're going to need a profile file - and I'm not convinced of that -
is there any reason to not always include it in the backup?

> Any comment will be appreciated. In particular I'd appreciate comments
> on correctness of relnode files detection and LSN extraction code.

I didn't look at it in detail, but one future problem comes to mind:
Once you implement the server-side code that only sends a file if its
LSN is higher than the cutoff point that the client gave, you'll have to
scan the whole file first, to see if there are any blocks with a higher
LSN. At least until you find the first such block. So with a file-level
implementation of this sort, you'll have to scan all files twice, in the
worst case.

- Heikki