Re: Releasing memory during External sorting?

On Fri, Sep 23, 2005 at 06:39:35PM +0200, Pailloncy Jean-Gerard wrote:
> >On most platforms it's quite unlikely that any memory would actually
> >get released back to the OS before transaction end, because the
> >memory blocks belonging to the tuplesort context will be intermixed
> >with blocks belonging to other contexts. So I think this is pretty
> >pointless. (If you can't afford to have the sort using all of
> >sort_mem, you've set sort_mem too large, anyway.)

> On OpenBSD 3.8 malloc use mmap(2) and no more sbrk.
> So, as soon as the bloc is free, it returns to the OS.
> Access to the freed pointer crashs immediatly.

Interesting point. Glibc also uses mmap() but only for allocations
greater than a few K, otherwise it's a waste of space.

I guess you would have to look into the postgresql allocator to see if
it doesn't divide the mmap()ed space up between multiple contexts.
Large allocations certainly appear to be passed off to malloc() but I
don't think execSort allocates all it's space in one go, it just counts
the space allocated by palloc().

So, unless someone goes and adds changes the tuplesort code to allocate
big blocks and use them only for tuples, I think you're going to run
into issues with data interleaved, meaning not much to give back to the
OS...
--
Martijn van Oosterhout <kleptog(at)svana(dot)org> http://svana.org/kleptog/
> Patent. n. Genius is 5% inspiration and 95% perspiration. A patent is a
> tool for doing 5% of the work and then sitting around waiting for someone
> else to do the other 95% so you can sue them.