Re: Multiple sorts in a query

On Tue, May 19, 2009 at 12:32 PM, Simon Riggs <simon(at)2ndquadrant(dot)com> wrote:
>
> If we have a query that uses multiple sorts, we may have a top-level
> sort, with child nodes that contain sorts also. In some cases we may
> find with sub-nodes that have both inner and outer sub-trees that
> contain sorts also.

Well a top-level sort and a child sort wouldn't both be accumulating
rows at the same time. The child could still be alive behaving like a
tuplestore though.

> If we allocate large chunks of memory we use malloc(). So complex
> queries can have multiple mallocs, followed by multiple reallocs. That
> in itself seems likely to end up with roughly double memory use, since
> realloc won't work properly/quickly with multiple mallocs. (Double since
> we allocate X bytes, then 2X bytes etc until we hit the limit.)

I think it's even worse than that since the old and new allocation
have to briefly coexist. So at least transiently we use 3x the size of
the actual array.

> When we later free() the memory, do we always free() it in the reverse
> order in which it was allocated? If not, how does that effect reducing
> the sbrk point, or other aspects of reusing allocated memory?
>
> Is it possible that Solaris's default malloc isn't appropriate for
> repeated use in complex queries that use multiple sorts?

Well anything's possible. Do you have any specific ideas? I would
expect any decent malloc library to shrink sbrk based on statically
analyzing where its allocations actually are, so I wouldn't expect the
pattern of frees to matter on that front. It might still fragment
memory if we allocate a bunch of large tuplestore/tuplesorts and then
allocate one object in a longer lived memory context.

What problems have you seen?

--
greg