Re: Re: [COMMITTERS] pgsql: Map basebackup tablespaces using a tablespace_map file

On Tue, Jun 9, 2015 at 3:29 PM, Amit Kapila <amit(dot)kapila16(at)gmail(dot)com> wrote:
> On Tue, Jun 9, 2015 at 10:56 AM, Fujii Masao <masao(dot)fujii(at)gmail(dot)com> wrote:
>>
>> On Tue, Jun 9, 2015 at 1:04 PM, Amit Kapila <amit(dot)kapila16(at)gmail(dot)com>
>> wrote:
>> > On Tue, Jun 9, 2015 at 9:09 AM, Fujii Masao <masao(dot)fujii(at)gmail(dot)com>
>> > wrote:
>> >>
>> >> On Tue, May 12, 2015 at 10:42 PM, Andrew Dunstan <andrew(at)dunslane(dot)net>
>> >> wrote:
>> >> > Map basebackup tablespaces using a tablespace_map file
>> >> >
>> >> > Windows can't reliably restore symbolic links from a tar format, so
>> >> > instead during backup start we create a tablespace_map file, which is
>> >> > used by the restoring postgres to create the correct links in
>> >> > pg_tblspc.
>> >> > The backup protocol also now has an option to request this file to be
>> >> > included in the backup stream, and this is used by pg_basebackup when
>> >> > operating in tar mode.
>> >> >
>> >> > This is done on all platforms, not just Windows.
>> >> >
>> >> > This means that pg_basebackup will not not work in tar mode against
>> >> > 9.4
>> >> > and older servers, as this protocol option isn't implemented there.
>> >>
>> >> While I was performing the recovery-related tests, I found that there
>> >> was
>> >> the case where $PGDATA/tablespace_map was not renamed to *.old at the
>> >> recovery.
>> >> Is this harmless and intentional?
>> >
>> > There shouldn't be any problem because we tablespace_map file
>> > only if backup file is present.
>> >
>> >> Sorry if this has been already
>> >> discussed so far.
>> >>
>> >> The steps to reproduce the problem is:
>> >>
>> >> 1. start base backup, i.e., call pg_start_backup().
>> >> 2. repeat some write transactions and checkpoints until the WAL file
>> >> containing
>> >> the checkpoint record that backup_label indicates will be removed.
>> >> 3. killall -9 postgres
>> >> 4. start the server and a crash recovery.
>> >>
>> >> At this time, the crash recovery fails with the following error
>> >> messages.
>> >> 2015-06-09 12:26:54 JST FATAL: could not locate required checkpoint
>> >> record
>> >> 2015-06-09 12:26:54 JST HINT: If you are not restoring from a backup,
>> >> try removing the file ".../backup_label".
>> >>
>> >> 5. according to the above hint message, remove $PGDATA/backup_label
>> >> and restart a crash recovery
>> >>
>> >> Then you can see that tablespace_map remains in $PGDATA after the
>> >> recovery
>> >> ends.
>> >>
>> >
>> > The basic idea is that tablespace_map file will be used in case we
>> > have to restore from a backup which contains tablespaces. So
>> > I think if user is manually removing backup_label, there is no
>> > meaning of tablespace_map, so that should also be removed. One
>> > way to address this is modify the Hint message suggesting that
>> > remove tablespace_map if present.
>> >
>> > Current :
>> > If you are not restoring from a backup, try removing the file
>> > \"%s/backup_label\
>> > Modify it to:
>> > If you are not restoring from a backup, try removing the file
>> > \"%s/backup_label\
>> > and, if present \"%s/tablespace_map\.
>>
>> Or what about removing tablespace_map file at the beginning of recovery
>> whenever backup_label doesn't exist?
>
> Yes, thats another way, but is it safe to assume that user won't need
> that file,

Is there really case where tablespace_map is necessary even though backup_label
doesn't exist? If not, it seems safe to get rid of the file when backup_label
is not present.

> I mean in the valid scenario (where both backup_label and
> tablespace_map are present and usable) also, we rename them to
> *.old rather than deleting it.

Yep, I'm OK to make the recovery rename the file to *.old rather than delete it.

> Yet another way could be we return error if tablespace_map is
> present whenever backup_label doesn't exist?

This needs another user operation, i.e., if a user wants to restart the server
at this case, he or she needs to remove (or rename) the file manually.

Regards,

--
Fujii Masao