pg_basebackup delays closing of stdout

From: Jeff Janes <jeff(dot)janes(at)gmail(dot)com>
To: pgsql-hackers <pgsql-hackers(at)postgresql(dot)org>
Subject: pg_basebackup delays closing of stdout
Date: 2019-07-24 02:16:26
Message-ID: CAMkU=1xiF0kxmCs_w7b1pp9G0pbS5-0FtOJrros3NpU21PWmaw@mail.gmail.com
Views: Raw Message | Whole Thread | Download mbox | Resend email
Thread:
Lists: pgsql-hackers

Ever since pg_basebackup was created, it had a comment like this:

* End of chunk. If requested, and this is the base tablespace
* write configuration file into the tarfile. When done, close the
* file (but not stdout).

But, why make the exception for output going to stdout? If we are done
with it, why not close it?

After a massive maintenance operation, I want to re-seed a streaming
standby, which I start to do by:

pg_basebackup -D - -Ft -P -X none | pxz > base.tar.xz

But the archiver is way behind, so when it finishes the basebackup part, I
get:

NOTICE: pg_stop_backup cleanup done, waiting for required WAL segments to
be archived
WARNING: pg_stop_backup still waiting for all required WAL segments to be
archived (60 seconds elapsed)
...

The base backup file is not finalized, because pg_basebackup has not closed
its stdout while waiting for the WAL segment to be archived. The file is
incomplete due to data stuck in buffers, so I can't copy it to where I want
and bring up a new streaming replica (which bypasses the WAL archive, so
would otherwise work). Also, if pg_basebackup gets interupted somehow while
it is waiting for WAL archiving, the backup will be invalid, as it won't
flush the last bit of data. Of course if it gets interupted, I would have
to test the backup to make sure it is valid. But testing it and finding
that it is valid is better than testing it and finding that it is not.

I think it makes sense for pg_basebackup to wait for the WAL to be
archived, but there is no reason for it to hold the base.tar.xz file
hostage while it does so.

If I simply remove the test for strcmp(basedir, "-"), as in the attached, I
get the behavior I desire, and nothing bad seems to happen. Meaning, "make
check -C src/bin/pg_basebackup/" still passes (but only tested on Linux).

Is there a reason not to do this?

Cheers,

Jeff

Attachment Content-Type Size
pg_basebackup_close_stdout.patch application/octet-stream 1022 bytes

Responses

Browse pgsql-hackers by date

  From Date Subject
Next Message Michael Paquier 2019-07-24 02:29:28 Re: pg_receivewal documentation
Previous Message David Rader 2019-07-24 01:59:52 Re: Procedure support improvements