Re: WAL archive cleanup om master

On Mon, Jan 06, 2014 at 07:16:25AM -0800, Adrian Klaver wrote:
> On 01/06/2014 03:18 AM, Martijn van Oosterhout wrote:
> >Hoi,
> >
> >I've setup a up WAL shipping configuration as described in the wiki. On
> >the master I have:
> >
> >archive_mode = on
> >archive_command = 'cp %p /path_to/archive/%f'
> >
> >and on the slave in the recovery.conf I have:
> >
> >archive_cleanup_command = 'pg_archivecleanup /path/to/archive/ %r'
> >restore_command = 'cp /path/to/archive/%f %p'
> >
> >In between the two machines I have an rsync process copying the files
> >from the master to the slave.
> >
> >My problem is, that the archive_cleanup_command on the slave is
> >clearing old WAL files, which the rsync on the master just copies again
> >because they are still on the master.
> >
> >I can make a script on the master that deletes files older than an
> >hour, but that will break horribly if the copying breaks for an hour.
> >
> >Is there a smarter way to do this, like having rsync not copy stuff
> >already copied once?
>
> So the rsync and archive_command combination is a belt and suspenders thing?

I'm not sure what you mean, isn't this the recommended way of doing
things? The configuration comes from here:

http://wiki.postgresql.org/wiki/Hot_Standby

The master saves the archives to a directory, rsync copies them to
the slave, where there restore_command can find them.

I suppose you could get the archive_command to rsync directly, but will
that work in the case of network failure?

Note that from a network topology view, the slave cannot connect to the
master, so streaming replication is out.

> As to rsync, if I understand what you want you might take a look at
> --existing:
>
> http://linux.die.net/man/1/rsync

I don't think that will help me, since the rsync is supposed to copy
the new WAL files. The problem is that it needs to *not* copy the old
ones.

However, right under that switch I see --remove-source-files which
looks closer, as long as I only have one slave that is.

Have a nice day,
--
Martijn van Oosterhout <kleptog(at)svana(dot)org> http://svana.org/kleptog/
> He who writes carelessly confesses thereby at the very outset that he does
> not attach much importance to his own thoughts.
-- Arthur Schopenhauer