Re: PITR Recovery

On Thu, 2004-06-17 at 22:47, Simon Riggs wrote:
> On Wed, 2004-06-16 at 02:49, Tom Lane wrote:
> > Simon Riggs <simon(at)2ndquadrant(dot)com> writes:
> > > -finalaction refers to what to do when target is reached - the purpose
> > > of this is to allow recovery of a database to occur when we don't have
> > > enough space for all of the xlogs at once, so we need to do recovery in
> > > batches.
> >
> > It seems to me that this is the only *essential* feature out of what
> > you've listed, and the others are okay to add later. So I question
> > your priorities:
> >
> > > In time for beta freeze, I think it is possible to do a limited subset
> > > of the above:
> > > - implement DATABASE only (whole instance, not specific database)
> > > - implement END OF LOGS and TO TIMESTAMP
> > > - implement THEN START only
> > > - implement using simple C, rather than bison
> >
> > which seem to include everything except the one absolute must-have
> > for any serious installation.
> >
>
> OK. At first, I disagreed, for many reasons.
>
> I discussion with Bruce, I believe a fairly neat streaming solution is
> possible.
>
> During recovery, as each request for a new xlog is made, we can make a
> system(3) call to a user defined recovery_program to retrieve the next
> xlog and out it in place. As each xlog is closed the file will be
> removed. The result of this would be to stream the data files through
> recovery, so no more than 1-2 files would ever be required to perform
> what could be (and is touted as this by other vendors) an infinite
> recovery.
>
> The result is that a backup tape (or other tape silo) could stream data
> straight through to recovery, and would completely circumvent and
> concern about insufficient disk space for recovery.
>
> This would involve changes to XLogFileOpen() in xlog.c and far less
> complex than I had imagined such functionality could be.
>
> This could be specified to PostgreSQL by using:
> - restore_program='cp %s %s' or similar
>
> I'll work more on the design, but not tonight.
>

Technically straightforward, though more complex I thought, but
streaming the xlog files during recovery works in prototype - great idea
Bruce and thanks for pushing for a solution in that area, Tom.
[It looks like we do need to have a separate command file dedicated to
recovery options, otherwise there's no way to tell difference between
crash and full media recovery - but I'll lose the pompous syntax.]

I'll include this (actually very few new/changed lines) and the xlog
refactoring (lots of moved lines, but few changes) in a single patch.

These changes are dependent upon, but otherwise independent of the PITR
Archival path submitted on 15th. If anybody has comments on that patch,
please pass them through ASAP, otherwise I may be building on sand.

My plan is to get this out ASAP (tonight, hopefully), then build on it
with a few extra tweaks, so we have a full set of options for PITR by
29th.

Thanks,

Best Regards, Simon Riggs