Re: WAL archiving and backup TAR

On Tue, Jun 23, 2009 at 10:18:30PM +0200, Jakov Sosic wrote:
> On Fri, 19 Jun 2009 09:43:28 -0600
> torrez <torrez(at)unavco(dot)org> wrote:
>
> > Hello,
> > I'm implementing WAL archiving and PITR on my production DB.
> > I've set up my TAR, WAL archives and pg_xlog all to be store on a
> > separate disk then my DB.
> > I'm at the point where i'm running 'Select pg_start_backup('xxx');'.
> >
> > Here's the command i've run for my tar:
> >
> > time tar -czf /pbo/podbackuprecovery/tars/pod-backup-$
> > {CURRDATE}.tar.gz /pbo/pod > /pbo/podbackuprecovery/pitr_logs/backup-
> > tar-log-${CURRDATE}.log 2>&1
> >
> > The problem is that this tar took just over 25 hours to complete. I
> > expected this to be a long process because since my DB is about 100
> > gigs.
> > But 25hrs seems a bit too long. Does anyone have any ideas how to
> > cut down on this time?
> >
> > Are there limitations to tar or gzip related to the size i'm working
> > with, or perhaps as a colleague suggested, tar/zip is a single
> > thread process and it may be bottlenecking one CPU (we run multiple
> > core). When I run top, gzip is running at about 12% of the CPU and
> > tar is around .4%. which adds up to 1/8 of 100% CPU, which number
> > wise one full CPU on our server since we have 8.
> >
> > After making the .conf file configurations I restarted my DB and
> > allowed normal transactions while I do the tar/zip.
> >
> > Your help is very much appreciated.
>
> Transfer it first and compress later. We have production db of around
> 170GB's and backup is around 2h to Tivoli Storage Manager server via
> ethernet (to IBM tape library).
>
> I would not prefer bzip over gzip, because it is less tested, and
> generaly you don't want your backup archive to have even minor sight of
> a possible doubt.... Production environment maybe, but backup never...
>

+1

The gzip step is holding up the copy the most. Another thing that
might be worth trying is the "star" program. It can use a shared
memory buffer to allow very rapid archiving.

Cheers,
Ken