Re: pg_basebackup for streaming base backups

On Tue, Jan 18, 2011 at 8:40 PM, Magnus Hagander <magnus(at)hagander(dot)net> wrote:
>> When I untar the tar file taken by pg_basebackup, I got the following
>> messages:
>>
>>    $ tar xf base.tar
>>    tar: Skipping to next header
>>    tar: Archive contains obsolescent base-64 headers
>>    tar: Error exit delayed from previous errors
>>
>> Is this a bug? This happens only when I create $PGDATA by using
>> initdb -X (i.e., I relocated the pg_xlog directory elsewhere than
>> $PGDATA).
>
> Interesting. What version of tar and what platform? I can't reproduce
> that here...

$ cat /etc/redhat-release
Red Hat Enterprise Linux Server release 5.4 (Tikanga)

$ uname -a
Linux hermes 2.6.18-164.el5 #1 SMP Tue Aug 18 15:51:48 EDT 2009 x86_64
x86_64 x86_64 GNU/Linux

$ tar --version
tar (GNU tar) 1.15.1

>> +               r = PQgetCopyData(conn, &copybuf, 0);
>> +               if (r == -1)
>>
>> Since -1 of PQgetCopyData might indicate an error, in this case,
>> we would need to call PQgetResult?.
>
> Uh, -1 means end of data, no? -2 means error?

The comment in pqGetCopyData3 says

/*
* On end-of-copy, exit COPY_OUT or COPY_BOTH mode and let caller
* read status with PQgetResult(). The normal case is that it's
* Copy Done, but we let parseInput read that. If error, we expect
* the state was already changed.
*/

Also the comment in getCopyDataMessage says

/*
* If it's a legitimate async message type, process it. (NOTIFY
* messages are not currently possible here, but we handle them for
* completeness.) Otherwise, if it's anything except Copy Data,
* report end-of-copy.
*/

So I thought that. BTW, walreceiver has already done that.

>> ReceiveTarFile seems refactorable by using GZWRITE and GZCLOSE
>> macros.
>
> You mean the ones from pg_dump? I don't think so. We can't use
> gzwrite() with compression level 0 on the tar output, because it will
> still write a gz header. With pg_dump, that is ok because it's our
> format, but with a .tar (without .gz) I don't think it is.

Right. I withdrow the comment.

>> +       /*
>> +        * Make sure we're unpacking into an empty directory
>> +        */
>> +       verify_dir_is_empty_or_create(current_path);
>>
>> Can pg_basebackup take a backup of $PGDATA including a tablespace
>> directory, without an error? The above code seems to prevent that....
>
> Uh, how do you mean it woul dprevent that? It requires that the
> directory you're writing the tablespace to is empty or nonexistant,
> but that shouldn't prevent a backup, no? It will prevent you from
> overwriting things with your backup, but that's intentional - if you
> don't need the old dir, just remove it.

What I'm worried about is the case where a tablespace is created
under the $PGDATA directory. In this case, ISTM that pg_basebackup
takes the backup of $PGDATA including the tablespace directory first,
and then takes the backup of the tablespace directory again. But,
since the tablespace directory is not already empty, the backup of
the tablespace seems to fail.

> Was easy, done with "-c <fast|slow>".

Thanks a lot!

Regards,

--
Fujii Masao
NIPPON TELEGRAPH AND TELEPHONE CORPORATION
NTT Open Source Software Center