Re: pg_basebackup delays closing of stdout

Hi,

On 2019-07-23 22:16:26 -0400, Jeff Janes wrote:
> Ever since pg_basebackup was created, it had a comment like this:
>
> * End of chunk. If requested, and this is the base tablespace
> * write configuration file into the tarfile. When done, close the
> * file (but not stdout).
>
> But, why make the exception for output going to stdout? If we are done
> with it, why not close it?

I think closing stdout is a bad idea that can cause issues in a lot of
situations. E.g. because a later open() will then use that fd (the
lowest unused fd always gets used), and then the next time somebody
wants to write something to stdout, there's normal output interspersed
with some random file. You'd at the least have to reopen /dev/null into
its place or such.

It also seems likely to be a trap for some future feature additions that
want to write another file to stdout or such - in contrast to the normal
files it can't be reopened.

> After a massive maintenance operation, I want to re-seed a streaming
> standby, which I start to do by:
>
> pg_basebackup -D - -Ft -P -X none | pxz > base.tar.xz
>
> But the archiver is way behind, so when it finishes the basebackup part, I
> get:
>
> NOTICE: pg_stop_backup cleanup done, waiting for required WAL segments to
> be archived
> WARNING: pg_stop_backup still waiting for all required WAL segments to be
> archived (60 seconds elapsed)
> ...
>
> The base backup file is not finalized, because pg_basebackup has not closed
> its stdout while waiting for the WAL segment to be archived. The file is
> incomplete due to data stuck in buffers, so I can't copy it to where I want
> and bring up a new streaming replica (which bypasses the WAL archive, so
> would otherwise work).

That seems more like an argument for sticking a fflush() there, than
closing stdout.

Greetings,

Andres Freund