Re: checkpoint_timeout and archive_timeout

Jeff Janes <jeff(dot)janes(at)gmail(dot)com> wrote:
> Kevin Grittner <Kevin(dot)Grittner(at)wicourts(dot)gov> wrote:

>> We pipe the file into pg_clearxlogtail | gzip and pipe it out to
>> the archive directory (with a ".gz" suffix), rather than using cp
>> and processing it later. Well, actually, we pipe it to a
>> directory on the same mount point as the archive directory and mv
>> it into place, as part of our scheme to avoid problems with
>> partial files.
>
> Do you have an example of that which you could share?
>
> I've run into two problems I'm trying to overcome. One is that
> pg_clearxlogtail fails on file formats it doesn't recognize but is
> asked to archive anyway, such as
> '000000010000000200000065.00000020.backup', for example. Perhaps
> it could just issue a warning and then pass the unrecognized file
> through unchanged, instead of bailing out with a fatal error.
>
> The other is that a pipeline in bash reports success even if an
> interior member of it failed. I think I know how to fix that
> under an actual shell script (as opposed to a pipeline stuffed
> into "archive_command"), but would like to see how other people
> have dealt with it.

I make no claims that this couldn't be improved upon, but it has
been humming along without any problems without modification for
about five years now on 80-some machines. Our archive_command is
just a call to a bash script that looks like this:

walPathname=$1
walFilename=$2

tempfile=/var/pgsql/data/wal-files-temp/$walFilename.gz
targetfile=/var/pgsql/data/wal-files/$walFilename.gz
errfile=/home/ccsa/archive.err
emailfile=/home/ccsa/$walFilename.rpt

cat <<END > runtime
walPathname=$walPathname
walFilename=$walFilename
tempfile=$tempfile
targetfile=$targetfile
errfile=$errfile
emailfile=$emailfile
END

if [ -f $targetfile ] ; then
echo "Cannot copy $walPathName to $targetfile \
because it already exists" 2> $errfile
archivereturnval=3
else
if echo $walFilename | grep -Eq ^[0-9A-F]{24}$ - ; then
pg_clearxlogtail < $walPathname 2> $errfile \
| gzip > $tempfile 2>> $errfile \
&& mv $tempfile $targetfile 2>> $errfile
archivereturnval=$?
else
gzip < $walPathname > $tempfile 2> $errfile \
&& mv $tempfile $targetfile 2>> $errfile
archivereturnval=$?
fi
fi

if [ "$archivereturnval" -eq "0" ] ; then
# Successful archive.
# Delete any email file from a possible prior failure.
rm -f $emailfile $errfile
exit 0
fi

# Failed to archive.
if [ ! -f $emailfile ] ; then
# No existing email files -- make one and send it.
echo "date: $(date +%F_%H%M)" > $emailfile
echo "to: dba(dot)group(at)wicourts(dot)gov" >> $emailfile
echo "subject: Problem with archiving $walFilename ">> $emailfile
echo " " >> $emailfile
echo "$archivereturnval return code from archive" >> $emailfile
echo " " >> $emailfile
cat $errfile >> $emailfile
echo " " >> $emailfile
cat runtime >> $emailfile
/usr/sbin/sendmail -t < $emailfile
fi
cat $errfile > /dev/stderr
exit $archivereturnval