Re: (full) Memory context dump considered harmful

Tomas Vondra <tomas(dot)vondra(at)2ndquadrant(dot)com> writes:
> On 08/21/2015 08:37 PM, Tom Lane wrote:
>> ... But suppose we add a parameter to memory context stats
>> collection that is the maximum number of child contexts to print *per
>> parent context*. If there are more than that, summarize the rest as per
>> your suggestion.
>>
>> The case where you would lose important data is where the serious
>> bloat is in some specific child context that is after the first N
>> children of its direct parent. I don't believe I've ever seen a case
>> where that was critical information as long as N isn't too tiny.

> Couldn't we make it a bit smarter to handle even cases like this? For
> example we might first count/sum the child contexts, and then print
> either all child contexts (if there are only a few of them) or just
> those that are >5% of the total, 2x the average or something like that.

That seems overly complicated. In the first place, you couldn't possibly
implement that without two passes over the context set, which would be
mighty expensive. (In the situations where this is getting run, most
likely portions of the address space have been swapped out, and we'll have
to swap them back in again. Bad enough to iterate over the whole process
address space once, but twice?) In the second place, it would seem to
require a much wider API between MemoryContextStats and the
per-context-type stats methods, including for example a way to decide
which numbers about a context were the most important ones. I'm already
concerned about having had to expose a set of accumulatable numbers at all
... don't want the API to get into their semantics in that kind of detail.

As I said, based on my experience in looking at memory context dumps
(and I've seen a few), a show-the-first-N-children heuristic probably
will work fine. If we find out differently we can work on refining it,
but I don't think a complex design is warranted at this stage.

One thing we could consider doing to improve the odds that it's fine
would be to rearrange things so that child contexts of the same parent
are more likely to be "similar" --- for example, break out all relcache
entries to be children of a RelCacheContext instead of the generic
CacheMemoryContext, likewise for cached plans, etc. But I'm not yet
convinced that'd be worth the trouble.

regards, tom lane