Re: Parallel leader process info in EXPLAIN

On Wed, Oct 30, 2019 at 10:39:04AM -0700, Peter Geoghegan wrote:
>On Wed, Oct 30, 2019 at 9:24 AM Melanie Plageman
><melanieplageman(at)gmail(dot)com> wrote:
>> Checked out the patches a bit and noticed that the tuplesort
>> instrumentation uses spaceUsed and I saw this comment in
>> tuplesort_get_stats()
>
>> might it be worth trying out the memory accounting API
>> 5dd7fc1519461548eebf26c33eac6878ea3e8788 here?
>
>I made exactly the same suggestion several years back, not long after
>the API was first proposed by Jeff. However, tuplesort.c has changed a
>lot since that time, to the extent that that comment now seems kind of
>obsolete. These days, availMem accounting still isn't used at all for
>disk sorts. Rather, the slab allocator is used. Virtually all the
>memory used to merge is now managed by logtape.c, with only fixed
>per-tape memory buffers within tuplesort.c. This per-tape stuff is
>tiny anyway -- slightly more than 1KiB per tape.
>
>It would now be relatively straightforward to report the memory used
>by disk-based sorts, without needing to use the memory accounting API.
>I imagine that it would report the high watermark memory usage during
>the final merge. It's possible for this to be lower than the high
>watermark during initial run generation (e.g. because of the
>MaxAllocSize limit in buffer size within logtape.c), but that still
>seems like the most useful figure to users. There'd be a new
>"LogicalTapeSetMemory()" function to go along with the existing
>LogicalTapeSetBlocks() function, or something along those lines.
>
>Not planning to work on this now, but perhaps you have time for it.
>

Another thing worth mentioning is that the memory accounting API does
nothing about the pfree() calls, mentioned in the comment. The memory is
tracked at the block level, so unless the pfree() frees the whole block
(which only really happens for oversized chunks) the accounting info
does not change.

regards

--
Tomas Vondra http://www.2ndQuadrant.com
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services