Re: dynamically allocating chunks from shared memory

Hi,

On 08/09/2010 05:02 PM, Bruce Momjian wrote:
> [ Sorry to be jumping into this thread late.]

No problem at all.

> I am not sure threads would greatly help us.

Note that I'm absolutely, certainly not advocating the use of threads
for Postgres.

> The major problem is that
> all of our our structures are currently contiguous in memory for quick
> access. I don't see how threading would help with that. We could use
> realloc(), but we can do the same in shared memory if we had a chunk
> infrastructure, though concurrent access to that memory would hurt us in
> either threads or shared memory.

I don't quite follow what you are trying to say here. Whether or not
structures are contiguous in memory might affect performance, but I
don't see the relation to programmer's habits and/or knowledge.

With our process-based design, the default is private memory (i.e. not
shared). If you need shared memory, you must specify a certain amount in
advance. That chunk of shared memory then is reserved and can't ever be
used by another subsystem. Even if you barely ever need that much shared
memory for the subsystem in question.

That's opposed to what lots of people are used to with the threaded
approach, where shared memory is the default. And where you can easily
and dynamically allocate *shared* memory. Whatever chunk of shared
memory one subsystem doesn't need is available to another one (modulo
fragmentation of the dynamic allocator, perhaps, but..)

> Fundamentally, recreating the libc memory allocation routines is not
> that hard.

Uh.. well, writing a good, scalable, dynamic allocator certainly poses
some very interesting problems. Writing one that doesn't violate any
patent or other IP as an additional requirement seems like a pretty
tough problem to me.

> (Everyone has to detach from the shared memory segment, but
> they have to stop using it too, so it doesn't seem that hard.)

So far, I only considered dynamically allocating from a pool of shared
memory that's initially fixed in size. So as to be able to make better
use of shared memory.

Resizing the overall pool the easy way, requiring every backend to
detach would cost a lot of performance. So that's certainly not
something you want to do often.

The purpose of such a dynamic allocator as I see it rather is to be able
to re-allocate unused memory of one subsystem to another one *on the
fly*. Not just for performance, but also for ease of use for the admin
and the developer, IMO.

Regards

Markus Wanner