Re: Seg fault when processing large SPI cursor (PG9.13)

"Fields, Zachary J. (MU-Student)" <zjfe58(at)mail(dot)missouri(dot)edu> writes:
> Thanks for getting back to me! I had already discovered freeing the SPI_tuptable each time, and you are correct, it made a big difference. However, I still was only able to achieve 140+ million before it crashed.

> My current working implementation is to reset the "current" memory context after X number of iterations, which keeps memory in check. This seems like a big hammer for the job, and I'm sure it is not optimal. Speed is very important to my application, so I would prefer to use a scalpel instead of a hatchet. If I am already freeing the SPI_tuptable created by the cursor, where else should I be looking for memory leaks?

There are a lot of places that could be leaking memory, for instance if
the array you're working with are large enough then they could be
toasted, and DatumGetArrayTypeP would involve making a working copy.
I'm not too sure that you're not ever leaking "candidate" structs,
either.

The usual theory in Postgres is that a memory context reset is cheaper,
as well as much less leak-prone, than trying to make sure you've pfree'd
each individual allocation. So we tend to work with short-lived
contexts that can be reset at the end of each tuple cycle --- or in this
example, probably once per cursor fetch would be good. The main problem
I'd have with what you're doing is that it's not very safe for a
function to reset the whole SPI Proc context: you might be clobbering
some storage that's still in use, eg related to the cursor you're using.
Instead create a context that's a child of that context, switch into
that to do your processing, and reset it every so often.

regards, tom lane