Re: [ANNOUNCE] IMCS: In Memory Columnar Store for PostgreSQL

On Thu, Jan 9, 2014 at 4:48 PM, Claudio Freire <klaussfreire(at)gmail(dot)com> wrote:
> On Thu, Jan 9, 2014 at 4:39 PM, knizhnik <knizhnik(at)garret(dot)ru> wrote:
>>> At fork time I only wrote about reserving the address space. After
>>> reserving it, all you have to do is implement an allocator that works
>>> in shared memory (protected by a lwlock of course).
>>>
>>> In essence, a hypothetical pg_dsm_alloc(region_name) would use regular
>>> shared memory to coordinate returning an already mapped region (same
>>> address which is guaranteed to work since we reserved that region), or
>>> allocate one (within the reserved address space).
>>
>> Why do we need named segments? There is ShmemAlloc function in PostgreSQL
>> API.
>> If RequestAddinShmemSpace can be used without requirement to place module in
>> preloaded list, then isn't it enough for most extensions?
>> And ShmemInitHash can be used to maintain named regions if it is needed...
>
> If you want to dynamically create the segments, you need some way to
> identify them. That is, the name. Otherwise, RequestWhateverShmemSpace
> won't know when to return an already-mapped region or not.
>
> Mind you, the name can be a number. No need to make it a string.
>
>> So if we have some reserved address space, do we actually need some special
>> allocator for this space to allocate new segments in it?
>> Why existed API to shared memory is not enough?

Oh, I notice why the confusion now.

The "reserve" mapping I was proposing, was a MAP_NORESERVE with PROT_NONE.

Ie: forbidden access. Which guarantees the OS won't try to allocate
physical RAM to it.

You'd have to re-map it before using, so it's not like a regular
shared memory region where you can simply allocate pointers and
intersperse bookkeeping data in-place.