Re: Backup history file should be replicated in Streaming Replication?

Robert Haas wrote:
> On Fri, Dec 18, 2009 at 12:22 PM, Florian Pflug <fgp(dot)phlo(dot)org(at)gmail(dot)com> wrote:
>> On 18.12.09 17:05 , Robert Haas wrote:
>>> On Fri, Dec 18, 2009 at 11:03 AM, Heikki Linnakangas
>>> <heikki(dot)linnakangas(at)enterprisedb(dot)com> wrote:
>>>> Or some way for to register the standby with the master so that
>>>> the master knows it's out there, and still needs the logs, even when
>>>> it's not connected.
>>> That is the real answer, I think.
>> It'd prefer if the slave could automatically fetch a new base backup if it
>> falls behind too far to catch up with the available logs. That way, old logs
>> don't start piling up on the server if a slave goes offline for a long time.
>>
>> The slave could for example run a configurable shell script in that case,
>> for example. You could then use that to rsync the data directory from the
>> server (after a pg_start_backup() of course).
>
> That would be nice to have too,

Yeah, for small databases, it's probably a better tradeoff. The problem
with keeping WAL around in the master indefinitely is that you will
eventually run out of disk space if the standby disappears for too long.

> but it's almost certainly much harder
> to implement. In particular, there's no hard and fast rule for
> figuring out when you've dropped so far behind that resnapping the
> whole thing is faster than replaying the WAL bit by bit.

I'd imagine that you take a new base backup only if you have to, ie. the
old WAL files the slave needs have already been deleted from the master.

> And, of
> course, you'll have to take the standby down if you go that route,
> whereas trying to catch up the WAL lets it stay up throughout the
> process.

Good point.

> I think (as I did/do with Hot Standby) that the most important thing
> here is to get to a point where we have a reasonably good feature that
> is of some use, and commit it. It will probably have some annoying
> limitations; we can remove those later. I have a feel that what we
> have right now is going to be non-robust in the face of network
> breaks, but that is a problem that can be fixed by a future patch.

Agreed. About a year ago, I was vocal about not relying on the file
based shipping, but I don't have a problem with relying on it as an
intermediate step, until we add the other options. It's robust as it is,
if you set up WAL archiving.

--
Heikki Linnakangas
EnterpriseDB http://www.enterprisedb.com