Re: WAL -> Replication

On Fri, 2002-04-26 at 19:41, Tom Lane wrote:
> Hannu Krosing <hannu(at)tm(dot)ee> writes:
> > DB2 can run in two modes
> > 1) similar to ours, where logs are reused after checkpoints/commits
> > allow it.
> > 2) with log archiving: logs are never reused, but when system determines
> > it no longer needs them, it will hand said log over to archiving process
> > that will archive it (usually do a backup to some other place and then
> > delete it).
>
> There is in fact the skeleton of support in xlog.c for passing unwanted
> log segments over to an archiver, rather than recycling them. So far
> no one's done anything with the facility. I think the main problem is
> the one Bruce cited: because the WAL representation is tied to physical
> tuple locations and so forth, it's only useful to a slave that has an
> *exact* duplicate of the master's entire database cluster. That's not
> useless, but it's pretty restrictive.

It is probably the fastest way to creating functionality for a hot spare
database.

If we could ship the log changes even earlier than whole logs are
complete, we can get near-realtime backup server.

> It could be useful for incremental backup, though I'm not sure how
> efficient it is for the purpose. WAL logs tend to be pretty voluminous.

But if they contain enough repeated data they should compress quite
well.

> At the very least you'd probably want enough smarts in the archiver to
> strip out the page-image records.

If we aim for ability to restore the last known good state and not any
point of time in between, the archiving can be just playing back the
logs over sparse files + keeping record (bitmap or list) of pages that
have been updated and thus are really present in the file. Then doing
full restore would be just restoring some point of time online backup
plus copying over changed pages.

----------------
Hannu