Re: [ANNOUNCE] IMCS: In Memory Columnar Store for PostgreSQL

On Wed, Jan 8, 2014 at 2:39 PM, knizhnik <knizhnik(at)garret(dot)ru> wrote:
> I wonder what is the intended use case of dynamic shared memory?
> Is is primarly oriented on PostgreSQL extensions or it will be used also in
> PosatgreSQL core?

My main motivation is that I want to use it to support parallel query.
There is unfortunately quite a bit of work left to be done before we
can make that a reality, but that's the goal.

> May be I am wrong, but I do not see some reasons for creating multiple DSM
> segments by the same extension.

Right.

> And total number of DSM segments is expected to be not very large (<10). The
> same is true for synchronization primitives (LWLocks for example) needed to
> synchronize access to this DSM segments. So I am not sure if possibility to
> place locks in DSM is really so critical...
> We can just reserved some space for LWLocks which can be used by extension,
> so that LWLockAssign() can be used without RequestAddinLWLocks or
> RequestAddinLWLocks can be used not only from preloaded extension.

If you're doing all of this at postmaster startup time, that all works
fine. If you want to be able to load up an extension on the fly, then
it doesn't. You can only RequestAddinLWLocks() at postmaster start
time, not afterwards, so currently any extension that wants to use
lwlocks has to be loaded at postmaster startup time, or you're out of
luck.

Well. Technically we reserve something like 3 extra lwlocks that
could be assigned later. But relying on those to be available is not
very reliable, and also, 3 is not very many, considering that we have
something north of 32k core lwlocks in the default configuration.

> IMHO the main trouble with DSM is lack of guarantee that segment is always
> mapped to the same virtual address.
> Without such guarantee it is not possible to use direct (normal) pointers
> inside DSM.
> But there seems to be no reasonable solution.

Yeah, that basically sucks. But it's very hard to do any better. At
least on a 64-bit platform, there's an awful lot of address space
available, and in theory it ought to be possible to find a portion of
that address space that isn't in use by any Postgres process and have
all of the backends map the shared memory segment there. But there's
no portable way to do that, and it seems like it would require an
awful lot of IPC to achieve consensus on where to put a new mapping.

On non-Windows platforms, Noah had the idea that could reserve a large
chunk of address space mapped as PROT_NONE and then overwrite it with
mappings later as needed. However, I'm not sure how portable that is
or whether it'll cause performance consequences (like page table
bloat) if the space doesn't end up getting used (or if it does). And
unless you have an awful lot of space available, it's hard to be sure
that new mappings are going to fit. And then there's Windows.

It would be nice to have better operating system support for this.
For example, IIUC, 64-bit Linux has 128TB of address space available
for user processes. When you clone(), it can either share the entire
address space (i.e. it's a thread) or none of it (i.e. it's a
process). There's no option to, say, share 64TB and not the other
64TB, which would be ideal for us. We could then map dynamic shared
memory segments into the shared portion of the address space and do
backend-private allocations in the unshared part. Of course, even if
we had that, it wouldn't be portable, so who knows how much good it
would do. But it would be awfully nice to have the option.

I haven't given up hope that we'll some day find a way to make
same-address mappings work, at least on some platforms. But I don't
expect it to happen soon.

--
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company