same-address mappings vs. relative pointers

During development of the dynamic shared memory facility, Noah and I
spent a lot of time arguing about whether it was practical to ensure
that a dynamic shared memory segment got mapped at the same address in
every backend that used it. The argument went something like this:

Me: We'll never be able to make that work reliably.
Noah: But if we can't use pointers in the dynamic shared memory
segment, our lives will suck.
Me: Well, we probably don't really NEED to use data structures that
contain pointers all THAT much.
Noah: Are you nuts? Of course we need pointers. They're ubiquitous
and essential.
Me: Meh.

I felt somewhat vindicated when I finished the dynamic shared memory
message queuing patch (which I'm still hoping someone will review
sometime soon...?) since that constitutes a useful chunk of
functionality that doesn't care about pointers at all. And there are
surely other examples that fall into the same category; for example,
an lwlock doesn't contain any pointers today, so storing one in a
dynamic shared memory segment in an address that might vary from one
process to another ought to work OK, too. I think there was actually
a patch a few years ago that made everything use LWLock * rather than
LWLockId, which would allow considerably more flexibility in laying
out lwlocks in memory - so you could for example try to put the in the
same cache lines as the data they protect, or different cache lines
than other hot lwlocks - and would probably also be almost enough to
allow placing them in a dynamic shared memory segment, which would be
useful.

But I'm also learning painfully that this kind of thing only goes so
far. For example, I spent some time looking at what it would take to
provide a dynamic shared memory equivalent of palloc/pfree, a facility
that I feel fairly sure would attract a few fans. Well, for regular
palloc, we store a pointer to the memory context before the beginning
of the chunk, so that when you call pfree you can find the memory
context and stick the chunk on one of its free lists. So there are
two pointers there: the pointer to the context is a pointer, of
course, but so is the free list. Heh, heh.

As I see it, if we want to have facilities like this, we'll have to
either (1) make same-address mappings work for as many architectures
as possible and don't support these facilities on the remainder or (2)
use relative pointers instead of absolute pointers within dynamic
shared memory segments, which means a loss of performance, notational
clarity, and type-safety. We can also (3) adopt both approaches -
some facilities can use relative pointers, which will be portable
everywhere but annoying otherwise, and others can work only when
same-address mappings are supported. Or we can (4) adopt neither
approach, and confine ourselves to data structures that don't use
pointers.

I still have mixed feelings about the idea of same-address mappings.
On the one hand, on 64-bit architectures, there's a huge amount of
address space available. Assuming the OS does something even vaguely
sensible in terms of laying out the text, heap, stack, and shared
library mappings, there ought to be many petabytes of address space
that never gets touched, and it's hard to see why we couldn't find
some place in there to stick our stuff. But that could require quite
a bit of OS-specific knowledge about how memory gets laid out. One
idea that I think is originally Noah's, though I may be mutilating it,
is to create a very large PROT_NONE mapping in the postmaster and then
overwrite that mapping with the mapping for any dynamic shared memory
segments we subsequently want to create. In that way, we essentially
reserve the address space we want to use before the child is forked
and things start to diverge (due to memory allocations, additional
shared library loads, etc.). But I bet that on at least some
operating systems that will actually allocate memory, or at least
count toward the system's notion of overcommit, and that will be a
problem. So I don't have any really good idea for how to implement
this cleanly.

Now, on the other hand, as far as dynamic shared memory allocation and
freeing is concerned, there aren't really THAT many places where I
need a pointer, so using Size or uint64 or something to store an
offset instead is annoying, but I have an idea how to do this that
only uses pointers in a couple of places, so I think it can be made to
work. I am not sure how much complaint that will provoke, though.
And even if I do it, the first poor slob that wants to store a linked
list or so in dynamic shared memory is going to be unhappy if they
can't get a same-address mapping. Maybe that's OK; using linked lists
in shared memory might not be a great idea in the first place. I'm
sure there will be more than one person who wants to do it, though.

Any thoughts on what the least painful compromise is here?

--
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company