Re: pg_basebackup failed to read a file

From: Stephen Frost <sfrost(at)snowman(dot)net>
To: Ron <ronljohnsonjr(at)gmail(dot)com>
Cc: pgsql-general(at)lists(dot)postgresql(dot)org
Subject: Re: pg_basebackup failed to read a file
Date: 2018-08-14 18:27:23
Message-ID: 20180814182722.GX3326@tamriel.snowman.net
Views: Raw Message | Whole Thread | Download mbox | Resend email
Thread:
Lists: pgsql-general

Greetings,

* Ron (ronljohnsonjr(at)gmail(dot)com) wrote:
> On 08/14/2018 11:14 AM, Tom Lane wrote:
> >Mike Cardwell <mike(dot)cardwell(at)hardenize(dot)com> writes:
> >>pg_basebackup: could not get write-ahead log end position from server:
> >>ERROR:  could not open file "./postgresql.conf~": Permission denied
> >>Now, I know what this error means. There was a root owned file at
> >>"/var/lib/pgsql/10/data/postgresql.conf~" which contained an old
> >>version of our postgres config and was not readable by the postgres
> >>user. I'll delete this file and try again. However, in the mean time: I
> >>feel like it would be useful for pg_basebackup to check that it has
> >>read access to all of the existing files in the source directory at the
> >>start, before it begins it's copy.
> >That seems like a pretty expensive thing to do, if there are lots of
> >files ... and you'd still end up failing, so it's not moving the ball
> >very far.
>
> Why is checking a bunch of file permissions anywhere close to being as
> expensive as transferring 1.5TB over a WAN link?

One could argue that the cost would be bourn by everyone who is using
pg_basebackup and not just those users who are transferring 1.5TB over a
WAN link.

That said, pgbackrest always builds a full manifest by scanning all of
the directories, tablespaces, files, etc, and I can't recall anyone ever
complaining about it. Certainly, failing fast would be better than
failing after a lot of time has been spent.

Thanks!

Stephen

In response to

Browse pgsql-general by date

  From Date Subject
Next Message Peter J. Holzer 2018-08-14 18:28:09 Re: Duplicating data folder without tablespace, for read access
Previous Message Tom Lane 2018-08-14 18:16:15 Re: pg_upgrade with large pg_largeobject table