Re: Streaming base backups

2011/1/10 Magnus Hagander <magnus(at)hagander(dot)net>:
> On Sun, Jan 9, 2011 at 23:33, Cédric Villemain
> <cedric(dot)villemain(dot)debian(at)gmail(dot)com> wrote:
>> 2011/1/7 Magnus Hagander <magnus(at)hagander(dot)net>:
>>> On Fri, Jan 7, 2011 at 01:47, Cédric Villemain
>>> <cedric(dot)villemain(dot)debian(at)gmail(dot)com> wrote:
>>>> 2011/1/5 Magnus Hagander <magnus(at)hagander(dot)net>:
>>>>> On Wed, Jan 5, 2011 at 22:58, Dimitri Fontaine <dimitri(at)2ndquadrant(dot)fr> wrote:
>>>>>> Magnus Hagander <magnus(at)hagander(dot)net> writes:
>>>>>>> * Stefan mentiond it might be useful to put some
>>>>>>> posix_fadvise(POSIX_FADV_DONTNEED)
>>>>>>>   in the process that streams all the files out. Seems useful, as long as that
>>>>>>>   doesn't kick them out of the cache *completely*, for other backends as well.
>>>>>>>   Do we know if that is the case?
>>>>>>
>>>>>> Maybe have a look at pgfincore to only tag DONTNEED for blocks that are
>>>>>> not already in SHM?
>>>>>
>>>>> I think that's way more complex than we want to go here.
>>>>>
>>>>
>>>> DONTNEED will remove the block from OS buffer everytime.
>>>
>>> Then we definitely don't want to use it - because some other backend
>>> might well want the file. Better leave it up to the standard logic in
>>> the kernel.
>>
>> Looking at the patch, it is (very) easy to add the support for that in
>> basebackup.c
>> That supposed allowing mincore(), so mmap(), and so probably switch
>> the fopen() to an open() (or add an open() just for mmap
>> requirement...)
>>
>> Let's go ?
>
> Per above, I still don't think we *should* do this. We don't want to
> kick things out of the cache underneath other backends, and since we
> can't control that. Either way, it shouldn't happen in the beginning,
> and if it does, should be backed with proper benchmarks.
>
> I've committed the backend side of this, without that. Still working
> on the client, and on cleaning up Heikki's patch for grammar/parser
> support.

attached is a small patch fixing "-d basedir" when its called with an
absolute path.
maybe we can use pg_mkdir_p() instead of mkdir ?

>
> --
>  Magnus Hagander
>  Me: http://www.hagander.net/
>  Work: http://www.redpill-linpro.com/
>

--
Cédric Villemain               2ndQuadrant
http://2ndQuadrant.fr/     PostgreSQL : Expertise, Formation et Support