Re: Streaming base backups

On Sun, Jan 9, 2011 at 12:05, Hannu Krosing <hannu(at)2ndquadrant(dot)com> wrote:
> On 9.1.2011 10:44, Magnus Hagander wrote:
>>
>> On Sun, Jan 9, 2011 at 09:55, Hannu Krosing<hannu(at)2ndquadrant(dot)com>  wrote:
>>>
>>> On 7.1.2011 15:45, Magnus Hagander wrote:
>>>>
>>>> On Fri, Jan 7, 2011 at 02:15, Simon Riggs<simon(at)2ndquadrant(dot)com>
>>>>  wrote:
>>>>
>>>>> One very useful feature will be some way of confirming the number and
>>>>> size of files to transfer, so that the base backup client can find out
>>>>> the progress.
>>>>
>>>> The patch already does this. Or rather, as it's coded it does this
>>>> once per tablespace.
>>>>
>>>> It'll give you an approximation only of course, that can change,
>>>
>>> In this case you actually could send exact numbers, as you need to only
>>> transfer the files
>>>  up to the size they were when starting the base backup. The rest will be
>>> taken care of by
>>>  WAL replay
>>
>> It will still be an estimate, because files can get smaller, and even
>> go away completely.
>
> Sure. Just wanted to remind the fact that you don't need to send the tail
> part of the
> file which was added after the start of backup.

True - but that's a PITA to keep track of. We do this if the file
changes during the transmission of that *file*, since otherwise the
tar header would specify an incorrect size, but not through the whole
backup.

> And you can give the "worst case" estimate for space needed by base backup.
>
> OTOH, streaming the WAL files in parallel can still fill up all available
> space :P

Yeah. I don't think it's worth the extra complexity of having to
enumerate and keep records for every individual file before the
streaming starts.

--
 Magnus Hagander
 Me: http://www.hagander.net/
 Work: http://www.redpill-linpro.com/