Re: [ANNOUNCE] IMCS: In Memory Columnar Store for PostgreSQL

On 01/08/2014 10:51 PM, Robert Haas wrote:
> On Tue, Jan 7, 2014 at 10:20 PM, Amit Kapila <amit(dot)kapila16(at)gmail(dot)com> wrote:
>> On Tue, Jan 7, 2014 at 2:46 AM, Robert Haas <robertmhaas(at)gmail(dot)com> wrote:
>>> On Mon, Jan 6, 2014 at 4:04 PM, james <james(at)mansionfamily(dot)plus(dot)com> wrote:
>>>> The point remains that you need to duplicate it into every process that
>>>> might
>>>> want to use it subsequently, so it makes sense to DuplicateHandle into the
>>>> parent, and then to advertise that handle value publicly so that other
>>>> child
>>>> processes can DuplicateHandle it back into their own process.
>>> Well, right now we just reopen the same object from all of the
>>> processes, which seems to work fine and doesn't require any of this
>>> complexity. The only problem I don't know how to solve is how to make
>>> a segment stick around for the whole postmaster lifetime. If
>>> duplicating the handle into the postmaster without its knowledge gets
>>> us there, it may be worth considering, but that doesn't seem like a
>>> good reason to rework the rest of the existing mechanism.
>> I think one has to try this to see if it works as per the need. If it's not
>> urgent, I can try this early next week?
> Anything we want to get into 9.4 has to be submitted by next Tuesday,
> but I don't know that we're going to get this into 9.4.
>
I wonder what is the intended use case of dynamic shared memory?
Is is primarly oriented on PostgreSQL extensions or it will be used also
in PosatgreSQL core?
In case of extensions, shared memory may be needed to store some
collected/calculated information which will be used by extension functions.

The main advantage of DSM (from my point of view) comparing with existed
mechanism of preloaded extension is that it is not necessary to restart
server to add new extension requiring shared memory.
DSM segment can be attached or created by _PG_init function of the
loaded module.
But there will be not so much sense in this mechanism if this segment
will be deleted when there are no more processes attached to it.
So to make DSM really useful for extension it needs some mechanism to
pin segment in memory during all server/extension lifetime.

May be I am wrong, but I do not see some reasons for creating multiple
DSM segments by the same extension.
And total number of DSM segments is expected to be not very large (<10).
The same is true for synchronization primitives (LWLocks for example)
needed to synchronize access to this DSM segments. So I am not sure if
possibility to place locks in DSM is really so critical...
We can just reserved some space for LWLocks which can be used by
extension, so that LWLockAssign() can be used without
RequestAddinLWLocks or RequestAddinLWLocks can be used not only from
preloaded extension.

IMHO the main trouble with DSM is lack of guarantee that segment is
always mapped to the same virtual address.
Without such guarantee it is not possible to use direct (normal)
pointers inside DSM.
But there seems to be no reasonable solution.