Re: dynamically allocating chunks from shared memory

On Mon, Jul 26, 2010 at 12:51 PM, Markus Wanner <markus(at)bluegap(dot)ch> wrote:
>> Dynamically allocating out of a 2MB
>> segment gives up most of that flexibility.
>
> Absolutely, that's why I'd like to see other modules that use the dynamic
> allocator. The more the better.

Right, I agree. The problem is that I don't think they can. The
elephant in the room is shared_buffers, which I believe to be
typically BY FAR the largest consumer of shared memory. It would be
absolutely fantastic if we had a shared_buffers implementation that
could free up unused buffers when they're not needed, or add more when
required. But there are several reasons why I don't believe that will
ever happen. One, much of the code that uses shared_buffers relies on
shared_buffers being located at a fixed memory address on a contiguous
chunk, and it's hard to see how we could change that assumption
without sacrificing performance. Two, the overall size of the shared
memory arena is largely dependent on the size of shared_buffers, so
unless you also have the ability to resize the arena on the fly (which
is well-nigh to impossible with our current architecture, and maybe
with any architecture), resizing shared_buffers doesn't actually add
that much flexibility. Three, the need for shared buffers is elastic
rather than absolute: stealing a few shared buffers for a defined
purpose (like sending imessages) is perfectly reasonable, but it's
rarely going to be a good idea for the buffer manager to proactively
free up memory just in case some other part of the system might need
some. If you have a system that normally has 4GB of shared buffers
and some other module borrows 100MB and then returns it, the system
will just cache less data while that memory is in use and then start
right back up caching more again once it's returned. That's very
nice, and it's hard to see how else to achieve that result.

Of course, there are other parts of the system (a whole bunch of them)
that used shared memory also, and perhaps some of those could be
modified to use the dynamic allocator as well. But they're getting by
without it now, so maybe they don't really need it. The SLRU stuff, I
think, works more or less like shared buffers (so you have the same
set of issues) and I think most of the other users are allocating
small, fixed-size chunks.

--
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise Postgres Company