Re: Req. for some help with resolving the error "pg_basebackup: could not get WAL end position from server: FATAL: requested WAL segment"

On Mon, Dec 5, 2016 at 3:27 PM, Tharmarajah, Sam <
sambavan(dot)tharmarajah(at)usask(dot)ca> wrote:

> Hi John,
>
>
>
> Thank you for the feed back. Presently it is taking around 1hr to finish
> the backups. We have the wal_keep_segments set to 50. Is there way to
> adjust the segment size to be more than 16MB? At times I notice in the
> pg_xlog, there are 10-15 segments on an invocation point.
>
>
>
>
>
> Thanx,
>
> Sam
>
>
>
> *From:* John Scalia [mailto:jayknowsunix(at)gmail(dot)com]
> *Sent:* Monday, December 05, 2016 1:52 PM
> *To:* Tharmarajah, Sam <sambavan(dot)tharmarajah(at)usask(dot)ca>
> *Cc:* pgsql-admin(at)postgresql(dot)org
> *Subject:* Re: [ADMIN] Req. for some help with resolving the error
> "pg_basebackup: could not get WAL end position from server: FATAL:
> requested WAL segment"
>
>
>
> More than likely your keep_wal_segments setting is too low. How long is
> the pg_basebackup taking? Remember that segments are generated about every
> 5 minutes, so if the backup takes an hour, you need at least 12 segments
> stored. At 2 hours, you need 24 etc., I'd set the value to about 12.2
> segments/hour of backup.
>
> --
>
> Jay
>
>
>
> Sent from my iPad
>
>
> On Dec 5, 2016, at 1:57 PM, Tharmarajah, Sam <
> sambavan(dot)tharmarajah(at)usask(dot)ca> wrote:
>
> Hi
>
>
>
> I am a newbie to postgres. We are on postgres 9.1 running on RedHat 6.
> When we do the postgres backups we are noticing the following errors
>
>
>
> pg_basebackup: could not get WAL end position from server: FATAL:
> requested WAL segment 000000010000020B00000040 has already been removed
>
>
>
> But when I look into pg_xlog, I notice
>
>
>
> 000000010000020B00000040.00000118.backup
>
>
>
>
>
> May I know how to resolve this error? Are the backups done with these
> errors, recoverable?
>
>
>
>
>
> Thanks in advance for your support.
>
>
>
> Thanx,
>
> Sam
>
>
>
>
>
>
>
>
>
>
If you're using pgbasebackup just for creating consistent backups, use the
--xlog-method=strean (-Xs) to ensure you get all the WAL files necessary
for a consistent backup. This avoids having to set wal_keep_segments high
enough to account for the time length of your entire backup. Note you will
need an extra WAL stream for this setting, so ensure max_wal_senders is set
high enough on the master.

Now if you're going to be using this backup to create a slave, you must get
that slave up and running before the master recycles all the WAL files
necessary for replication to be able to catch up from when the backup
stopped. Otherwise you will see that error you shared in your original
post. That's when you may need to adjust wal_keep_segments. Either that or
you should set up an archive_command on the master to keep a secondary copy
of all WAL files.

Keith