Re: Regarding Postgres Dynamic Shared Memory (DSA)

Hi Thomas,

I'm implementing the In-Memory index as per your suggestion. So far it's good.

As of now, only one thing is unclear for me. How could i detach the dsa(dsa_detach() call) in backend (typically during backend quit).

Of-course when process quits, all it's associated memory will be cleared/destroyed. But we have to decrement dsm reference counts as part of the potgres dsa framework implementation. which can not be done now.

I have gone through the postgres source code. But unfortunately, i couldn't see anything in handling this case.

Thanks a lot.

Best Regards,

-Mahi
Teamwork divides the task and multiplies the success.

---- On Wed, 21 Jun 2017 13:13:28 +0530 Mahendranath Gurram <mahendranath(at)zohocorp(dot)com> wrote ----

Hi Thomas,

I like the whole idea.

In fact, i understood this in your very first response itself.

Only thing is every time i have to check for dsa_attached or not. I mean get_my_shared_state() is NULL or not.

To avoid that check, i tried creating it in _PG_Init(postmaster process itself) all that stuff :(

Anyways the solution you suggested will work for me. I'll live with that if check.

Thanks a lot for all your help.

I'll implement this and update the status.

Cheers :)

Thanks & Best Regards,

-Mahi
Teamwork divides the task and multiplies the success.

---- On Wed, 21 Jun 2017 12:02:30 +0530 Thomas Munro <thomas(dot)munro(at)enterprisedb(dot)com> wrote ----

On Wed, Jun 21, 2017 at 5:27 PM, Mahendranath Gurram

<mahendranath(at)zohocorp(dot)com> wrote:

> Initially i tried to design the same way.

> I mean, i have created a background worker and created dsa in it.

> I tried to attach/detach to the same dsa/dsm by all the backends(postgres

> clients/connections) during backend(client/connection) init/destroy.

> I didn't find any triggers or callbacks during backend init/close to

> attach/detach the dsa/dsm. Hence, i took this approach.

> If postgres have any such triggers/callbacks available, please let me know,

> that is of great great help for me.

>

> Anyways now i understood, i have taken a wrong approach to use dsa. I'll try

> to figure out any other way to build my in-memory index over postgres.

You definitely can use DSA or DSM for this. As a matter of fact

shared in-memory indexing was one of our target use cases. You just

have to jump through a few hoops... Here's one approach:

1. Use _PG_init and the shmem hook to reserve a little bit of

traditional shared memory and initialise it to zero. This will be

used just to share the DSA handle, but you can't actually create the

DSA area in postmaster. In other words, this little bit of shared

memory is for "discovery", since it can be looked up by name from any

backend.

2. In each backend that wants to use your new in-memory index system,

you need to be able to attach or create the DSA area on-demand.

Perhaps you could have a get_my_shared_state() function (insert better

name) that uses a static local variable to hold a pointer to some

state. If it's NULL, you know you need to create the state. That

should happen only once in each backend, the first time through the

function. In that case you need to create or attach to the DSA area

as appropriate, which you should wrap in

LWLockAcquire(AddinShmemInitLock,

LW_EXCLUSIVE)/LWLockRelease(AddinShmemInitLock) to serialise the code

block. First, look up the bit of traditional shared memory to see if

there is a DSA handle published in it already. If there is you can

attach. If there isn't, you are the first so you need to create, and

publish the handle for others to attach to. Remember whatever state

you need to remember, such as the dsa_area, in static local variables

so that all future calls to get_my_shared_state() in that backend will

be fast.

If you do it that way, then it doesn't matter whether it's background

workers or foreground processes: whoever is first to call

get_my_shared_state() will create the DSA area (and whatever else you

want to do to set things up). All later callers will attach to the

existing one. But each backend only ever has to enter the locked code

path once, and from then on it's fast and there's no lock.

Note that you could skip step 1 and not require a preload shared

library. Then you'd be creating the 'discovery' shmem region on

demand too! Just make sure it's in the locked region and pay

attention to the 'found' output variable to decide whether you're

first and need to initialise it. Skipping step 1 is very slightly

against the rules though, because you'd be using a little piece of

memory that you didn't tell the postmaster to reserve space for with

RequestAddinShmemSpace. It's very small though, so you might decide

that's OK...

If you don't like the idea of creating that shmem stuff on demand, you

could look into using session_preload_libraries as a way to get a

'backend init' hook.

--

Thomas Munro

http://www.enterprisedb.com

--

Sent via pgsql-hackers mailing list (pgsql-hackers(at)postgresql(dot)org)

To make changes to your subscription:

http://www.postgresql.org/mailpref/pgsql-hackers