Re: [FEATURE REQUEST] Streaming Onlinebackup (Maybe OFFTOPIC)

On Wed, 2007-09-26 at 11:05 -0500, Kevin Grittner wrote:

> Lacking any suggestions, I plowed ahead with something which satisfies
> our needs. First, rough, version attached. It'll save us buying another
> drawer of drives, so it was worth a few hours of research to figure out
> how to do it.
>
> If anyone spots any obvious defects please let me know. We'll be running
> about 50,000 WAL files through it today or tomorrow; if any problems turn
> up in that process I'll repost with a fix.
>
> Given the lack of response to my previous post, I'll assume it's not worth
> the effort to do more in terms of polishing it up; but if others are
> interested in using it, I'll make some time for that.
>
> Adding this to the pipe in our archive script not only saves disk space,
> but reduces the CPU time overall, since gzip usually has less work to do.
> When WAL files switch because they are full, the CPU time goes from about
> 0.8s to about 1.0s.

It's nicely written and looks like it would perform well.

The logic for zeroing the blocks makes me nervous. It doesn't locate the
block from which to start, it treats all blocks equally, so might zero
some blocks and not others. What you have should work, but I'd be
inclined to put a test in there to check that doesn't happen: once we
begin to zero pages, all of them should be zeroed to end of file. If we
find one that shouldn't be zeroed, throw an error.

We should also document that this is designed to help compress files
that aren't full because we switched early because of archive_timeout.

--
Simon Riggs
2ndQuadrant http://www.2ndQuadrant.com