pg_backrest no longer backing up

From: Aldo Sarmiento <aldo(at)bigpurpledot(dot)com>
To: pgsql-admin(at)postgresql(dot)org
Subject: pg_backrest no longer backing up
Date: 2017-03-08 19:48:48
Message-ID: CAHX=r6yti58x1z4y8q_zqr_P+8_Aq5_uPKSJDu1xDaV+rtjQmA@mail.gmail.com
Views: Raw Message | Whole Thread | Download mbox | Resend email
Thread:
Lists: pgsql-admin

Hello,

Sometime in December in 2016 we ran out of space for our backup directory,
which we increased promptly. However, it appears that backrest stopped
taking backups since the incident.

Versions of things:

pg_backrest 0.75
postgresql 9.4
ubuntu 14.04 (This is the backup server, not master)

Couple things I noticed:

*postgres(at)pgdedicated:/backup/backrest/log$ cat
prod-9.4-backup-20170308.log*
-------------------PROCESS START-------------------
2017-03-08 01:30:02.034 T00 INFO: backup start: type = diff
2017-03-08 01:30:02.835 T00 INFO: last backup label = 20161204-025729F,
version = 0.75
2017-03-08 01:30:02.911 T00 INFO: executing pg_start_backup() with label
"pg_backrest backup started 2017-03-08 01:30:02": backup will begin after
the requested immediate checkpoint completes
2017-03-08 01:30:02.976 T00 ERROR: process terminated on signal or
exception, 0 threads stopped

*postgres(at)pgdedicated:/backup/backrest/log$ ls -lah
/backup/backrest/backup/prod-9.4/*
total 44K
drwxrwxr-x 11 postgres postgres 4.0K Dec 22 14:02 .
drwxrwxr-x 3 postgres postgres 4.0K Mar 15 2016 ..
drwxrwxr-x 3 postgres postgres 4.0K Nov 20 03:03 20161120-030305F
drwxrwxr-x 3 postgres postgres 4.0K Nov 27 02:53 20161127-025355F
drwxrwxr-x 3 postgres postgres 4.0K Dec 3 02:24
20161127-025355F_20161203-022437D
drwxrwxr-x 3 postgres postgres 4.0K Dec 4 02:57 20161204-025729F
drwxrwxr-x 3 postgres postgres 4.0K Dec 5 02:24
20161204-025729F_20161205-022427D
drwxrwxr-x 3 postgres postgres 4.0K Dec 6 02:25
20161204-025729F_20161206-022512D
drwxrwxr-x 3 postgres postgres 4.0K Dec 7 02:28
20161204-025729F_20161207-022814D
drwxrwxr-x 3 postgres postgres 4.0K Dec 8 02:27
20161204-025729F_20161208-022730D
drwxrwxr-x 3 postgres postgres 4.0K Dec 9 02:28
20161204-025729F_20161209-022855D
lrwxrwxrwx 1 postgres postgres 33 Dec 9 02:28 latest ->
20161204-025729F_20161209-022855D

*postgres(at)pgdedicated:/backup/backrest/log$ cat /etc/cron.d/backrest*
# Vacuum every Sunday before Full backup
#05 00 * * 0 postgres vacuumdb -qaz

# Full backup every Sunday
#30 01 * * 0 postgres /usr/lib/backrest/bin/pg_backrest.pl
--stanza=prod-9.4 --type=full backup

# Differential backup all other days
#30 01 * * 1,2,3,4,5,6 postgres /usr/lib/backrest/bin/pg_backrest.pl
--stanza=prod-9.4 --type=diff backup

*postgres(at)pgdedicated:/backup/backrest/log$ cat /etc/pg_backrest.conf*
[global:general]
repo-path=/backup/backrest

[global:backup]
start-fast=y

[global:expire]
retention-full=6 *# This was originally 8, dialed it down to 6 when ran out
of space*
retention-diff=6
retention-archive-type=full
retention-archive=6 *# This was originally 8, dialed it down to 6 when ran
out of space*

[prod-9.4]
db-path=/var/lib/postgresql/9.4/replica

Question: Not sure when the backrest cron was commented out, but how is it
still attempting to do backups without this setup? I'm assuming it's being
invoked some other way?

Also, any idea what the issue is with the error?

Responses

Browse pgsql-admin by date

  From Date Subject
Next Message David Steele 2017-03-08 22:18:07 Re: pg_backrest no longer backing up
Previous Message Ray Stell 2017-03-08 17:43:28 Re: pg_basebackup recovery failure