Re: Allocating shared memory in Postgres

For starters, what happens if two backends do this concurrently?

> I'm assuming here that a single backend process handles all queries from
a given client. In case of parallel queries, the master process will be
responsible for allocating the shared memory and not the workers. Please
let me know if this is not something that you implied by two backends.

Even with only one backend, if you do a query that requires X space, and
then you do another query that requires X+1 space, what's going to happen?

> In my application, every query that allocates shared memory is wrapped in
a separate txn, and the life of the shared memory is for the duration of
the query. The only purpose of allocating shared memory here is to make the
memory segment visible to the worker processes for that particular query.
No other txns/query actually accesses this shared memory. Also, when I
allocate shared memory, the txn id is used as a key to the ShmemIndex to
differentiate between two concurrent shared memory allocation requests.

Recent PG releases have a "DSM" mechanism for short-lived (query lifespan,
typically) shared memory that's separate from the core shmem pool. That
might suit your needs better. The system design is really not friendly
to demanding more core shmem after postmaster start.

> Yes, I understand that "DSM" mechanisms exist. But I wanted to know if
the approach that I had outlined will work even if there are certain
drawbacks to it such as
1. overestimating the initial shared memory size that needs to be
allocated,
2. not able to free the shared memory after use

Best,
-SB

On Wed, Jul 3, 2019 at 4:17 PM Tom Lane <tgl(at)sss(dot)pgh(dot)pa(dot)us> wrote:

> Souvik Bhattacherjee <kivuosb(at)gmail(dot)com> writes:
> > I need to allocate shared memory in Postgres 11.0 outside the
> > initialization phase. In order to achieve this I have done the following:
> > - during a particular query, where I need to allocate shared memory
> (which
> > is a function of the sizes of the tables in the query), I invoke
> > ShmemInitStruct() in shmem.c
>
> This seems like a pretty horrid idea. For starters, what happens if two
> backends do this concurrently? Even with only one backend, if you do a
> query that requires X space, and then you do another query that requires
> X+1 space, what's going to happen?
>
> Recent PG releases have a "DSM" mechanism for short-lived (query lifespan,
> typically) shared memory that's separate from the core shmem pool. That
> might suit your needs better. The system design is really not friendly
> to demanding more core shmem after postmaster start.
>
> regards, tom lane
>