Re: Error: dsa_area could not attach to a segment that has been freed

Hi Thomas,

Thanks for cautioning us about possible memory leaks(during error cases) incase of long-lived DSA segements.

Actually we are following an approach to avoid this DSA memory leaks. Let me explain our implementation and please validate and correct us in-case we miss anything.

Implementation:

Basically we have to put our index data into memory (Index Column Value Vs Ctid) which we get in aminsert callback function.

Coming to the implementation, in aminsert Callback function,

We Switch to CurTransactionContext

Cache the DMLs of a transaction into dlist(global per process)

Even if different clients work parallel, it won't be a problem because every client gets one dlist in separate process and it'll have it's own CurTransactionContext

We have registered transaction callback (using RegisterXactCallback() function). And during event pre-commit(XACT_EVENT_PRE_COMMIT), we populate all the transaction specific DMLs (from dlist) into our in-memory index(DSA) obviously inside PG_TRY/PG_CATCH block.

In case we got some errors(because of dsa_allocate() or something else) while processing dlist(while populating in-memory index), we cleanup the DSA memory in PG_CATCH block that is allocated/used till that point.

During other error cases, typically transactions gets aborted and PRE_COMMIT event is not called and hence we don't touch DSA at that time. Hence no need to bother about leaks.

Even sub transaction case is handled with sub transaction callbacks.

CurTransactionContext(dlist basically) is automatically cleared after that particular transaction.

I want to know if this approach is good and works well in all cases. Kindly provide your feedback on this.

Regards

G. Sai Ram

---- On Wed, 20 Sep 2017 14:25:43 +0530 Thomas Munro <thomas(dot)munro(at)enterprisedb(dot)com> wrote ----

On Wed, Sep 20, 2017 at 6:14 PM, Gaddam Sai Ram

<gaddamsairam(dot)n(at)zohocorp(dot)com> wrote:

> Thank you very much! That fixed my issue! :)

> I was in an assumption that pinning the area will increase its lifetime but

> yeah after taking memory context into consideration its working fine!

So far the success rate in confusing people who first try to make

long-lived DSA areas and DSM segments is 100%. Basically, this is all

designed to ensure automatic cleanup of resources in short-lived

scopes.

Good luck for your graph project. I think you're going to have to

expend a lot of energy trying to avoid memory leaks if your DSA lives

as long as the database cluster, since error paths won't automatically

free any memory you allocated in it. Right now I don't have any

particularly good ideas for mechanisms to deal with that. PostgreSQL

C has exception-like error handling, but doesn't (and probably can't)

have a language feature like scoped destructors from C++. IMHO

exceptions need either destructors or garbage collection to keep you

sane. There is a kind of garbage collection for palloc'd memory and

also for other resources like file handles, but if you're using a big

long lived DSA area you have nothing like that. You can use

PG_TRY/PG_CATCH very carefully to clean up, or (probably better) you

can try to make sure that all your interaction with shared memory is

no-throw (note that that means using dsa_allocate_extended(x,

DSA_ALLOC_NO_OOM), because dsa_allocate itself can raise errors). The

first thing I'd try would probably be to keep all shmem-allocating

code in as few routines as possible, and use only no-throw operations

in the 'critical' regions of them, and maybe look into some kind of

undo log of things to free or undo in case of error to manage

multi-allocation operations if that turned out to be necessary.

--

Thomas Munro

http://www.enterprisedb.com