Re: Regarding Postgres Dynamic Shared Memory (DSA)

On Wed, Jun 21, 2017 at 5:27 PM, Mahendranath Gurram
<mahendranath(at)zohocorp(dot)com> wrote:
> Initially i tried to design the same way.
> I mean, i have created a background worker and created dsa in it.
> I tried to attach/detach to the same dsa/dsm by all the backends(postgres
> clients/connections) during backend(client/connection) init/destroy.
> I didn't find any triggers or callbacks during backend init/close to
> attach/detach the dsa/dsm. Hence, i took this approach.
> If postgres have any such triggers/callbacks available, please let me know,
> that is of great great help for me.
>
> Anyways now i understood, i have taken a wrong approach to use dsa. I'll try
> to figure out any other way to build my in-memory index over postgres.

You definitely can use DSA or DSM for this. As a matter of fact
shared in-memory indexing was one of our target use cases. You just
have to jump through a few hoops... Here's one approach:

1. Use _PG_init and the shmem hook to reserve a little bit of
traditional shared memory and initialise it to zero. This will be
used just to share the DSA handle, but you can't actually create the
DSA area in postmaster. In other words, this little bit of shared
memory is for "discovery", since it can be looked up by name from any
backend.

2. In each backend that wants to use your new in-memory index system,
you need to be able to attach or create the DSA area on-demand.
Perhaps you could have a get_my_shared_state() function (insert better
name) that uses a static local variable to hold a pointer to some
state. If it's NULL, you know you need to create the state. That
should happen only once in each backend, the first time through the
function. In that case you need to create or attach to the DSA area
as appropriate, which you should wrap in
LWLockAcquire(AddinShmemInitLock,
LW_EXCLUSIVE)/LWLockRelease(AddinShmemInitLock) to serialise the code
block. First, look up the bit of traditional shared memory to see if
there is a DSA handle published in it already. If there is you can
attach. If there isn't, you are the first so you need to create, and
publish the handle for others to attach to. Remember whatever state
you need to remember, such as the dsa_area, in static local variables
so that all future calls to get_my_shared_state() in that backend will
be fast.

If you do it that way, then it doesn't matter whether it's background
workers or foreground processes: whoever is first to call
get_my_shared_state() will create the DSA area (and whatever else you
want to do to set things up). All later callers will attach to the
existing one. But each backend only ever has to enter the locked code
path once, and from then on it's fast and there's no lock.

Note that you could skip step 1 and not require a preload shared
library. Then you'd be creating the 'discovery' shmem region on
demand too! Just make sure it's in the locked region and pay
attention to the 'found' output variable to decide whether you're
first and need to initialise it. Skipping step 1 is very slightly
against the rules though, because you'd be using a little piece of
memory that you didn't tell the postmaster to reserve space for with
RequestAddinShmemSpace. It's very small though, so you might decide
that's OK...

If you don't like the idea of creating that shmem stuff on demand, you
could look into using session_preload_libraries as a way to get a
'backend init' hook.

--
Thomas Munro
http://www.enterprisedb.com