Re: Forcing current WAL file to be archived

> > The problems I see with this is if in this case the normal postgres
> > WAL
> > archiving won't conflict with this streaming ?
>
> You are not forced to use it if your shell scripts do conflict.
>
> What I envisioned, was that the current WAL archiving shell script would
> just do some CRC check over the WAL's already shipped, or as we
> currently use rsync to do the actual shipping this is what happens
> automatically.

Hmm, that sounds pretty smart... the archive process rsyncing over the
file which was previously streamed... I guess this will mean very little
overhead (in fact it only means the WAL archive to be read once more
than absolutely necessary, and the CPU power to compute the CRCs).

> > And if yes, wouldn't it
> > be better to have a separate mechanism for the stream based
> > archiving ?
>
> why separate ? I'm a great believer in doing the minimum useful change,
> at least in systems used in production. We already have a working
> solution for full file shipping, so why not just augment it with
> streaming the currently-written-to file.

That's good so, I also have a working script, so I'm also not very
motivated to do anything more complicated... but 6 months ago I would
have been really glad to have a stand-alone program which I could
install along postgres on the slave, point it to the master, and get a
working WAL shipping based stand-by. Instead I spent a few days setting
up our standby scripts and testing it under load... and never being
certain it really works and it won't break exactly when I need it
most...

> > I mean what will happen if postgres successfully archives the WAL file
> > and then moves it out of way before the streaming process finishes
> > with
> > it, the streaming process will have a problem...
>
> This should not happen. your streaming process should be smart enought
> to guarantee that.

OK, true, the streaming script should always stream only the current
file. If the last offset was from a previous WAL, it can be safely reset
to 0, and stream the new WAL from the beginning. So the streaming script
needs to remember the last WAL and offset, not just the offset.

Cheers,
Csaba.