Re: pg_basebackup failed to read a file

Greetings,

* Ron (ronljohnsonjr(at)gmail(dot)com) wrote:
> On 08/14/2018 11:14 AM, Tom Lane wrote:
> >Mike Cardwell <mike(dot)cardwell(at)hardenize(dot)com> writes:
> >>pg_basebackup: could not get write-ahead log end position from server:
> >>ERROR:  could not open file "./postgresql.conf~": Permission denied
> >>Now, I know what this error means. There was a root owned file at
> >>"/var/lib/pgsql/10/data/postgresql.conf~" which contained an old
> >>version of our postgres config and was not readable by the postgres
> >>user. I'll delete this file and try again. However, in the mean time: I
> >>feel like it would be useful for pg_basebackup to check that it has
> >>read access to all of the existing files in the source directory at the
> >>start, before it begins it's copy.
> >That seems like a pretty expensive thing to do, if there are lots of
> >files ... and you'd still end up failing, so it's not moving the ball
> >very far.
>
> Why is checking a bunch of file permissions anywhere close to being as
> expensive as transferring 1.5TB over a WAN link?

One could argue that the cost would be bourn by everyone who is using
pg_basebackup and not just those users who are transferring 1.5TB over a
WAN link.

That said, pgbackrest always builds a full manifest by scanning all of
the directories, tablespaces, files, etc, and I can't recall anyone ever
complaining about it. Certainly, failing fast would be better than
failing after a lot of time has been spent.

Thanks!

Stephen