Re: [ANNOUNCE] IMCS: In Memory Columnar Store for PostgreSQL

On Fri, Jan 10, 2014 at 1:35 PM, Claudio Freire <klaussfreire(at)gmail(dot)com> wrote:
>>> You can map a segment at fork time, and unmap it after forking. That
>>> doesn't really use RAM, since it's supposed to be lazily allocated (it
>>> can be forced to be so, I believe, with PROT_NONE and MAP_NORESERVE,
>>> but I don't think that's portable).
>>>
>>> That guarantees it's free.
>>
>> It guarantees that it is free as of the moment you unmap it, but it
>> doesn't guarantee that future memory allocations or shared library
>> loads couldn't stomp on the space.
>
> You would only unmap prior to remapping, only the to-be-mapped
> portion, so I don't see a problem.

OK, yeah, that way works. That's more or less what Noah proposed
before. But I was skeptical it would work well everywhere. I suppose
we won't know until somebody tries it. (I didn't.)

> Ok, yes, fragmentation could be an issue if the address range is not
> "humongus enough".

I've often thought that 64-bit machines are so capable that there's no
reason to go any higher. But lately I've started to wonder. There
are already machines out there with >2^40 bytes of physical memory,
and the number just keeps creeping up. When you reserve a couple of
bits to indicate user or kernel space, and then consider that virtual
address space can be many times larger than physical memory, it starts
not to seem like that much.

But I'm not that excited about the amount of additional memory we'll
eat when somebody decides to make a pointer 16 bytes. Ugh.

--
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company