Re: Reducing System Allocator Thrashing of ExecutorState to Alleviate FDW-related Performance Degradations

On Sat, 18 Feb 2023 at 06:52, Andres Freund <andres(at)anarazel(dot)de> wrote:
> And did you increase ALLOCSET_DEFAULT_INITSIZE everywhere, or just passed a
> larger block size in CreateExecutorState()? If the latter,the context
> freelist wouldn't even come into play.

I think this piece of information is critical to confirm what the issue is.

> A 8MB max block size is pretty darn small if you have a workload that ends up
> with a gigabytes worth of blocks.

We should probably review that separately. These kinds of definitions
don't age well. The current ones appear about 23 years old now, so we
might be overdue to reconsider what they're set to.

2002-12-15 21:01:34 +0000 150) #define ALLOCSET_DEFAULT_MINSIZE 0
2000-06-28 03:33:33 +0000 151) #define ALLOCSET_DEFAULT_INITSIZE (8 * 1024)
2000-06-28 03:33:33 +0000 152) #define ALLOCSET_DEFAULT_MAXSIZE (8 *
1024 * 1024)

... I recall having a desktop with 256MBs of RAM back then...

Let's get to the bottom of where the problem is here before we
consider adjusting those. If the problem is unrelated to that then we
shouldn't be discussing that here.

> And the problem also could just be that the default initial blocks size takes
> too long to ramp up to a reasonable block size. I think it's 20 blocks to get
> from ALLOCSET_DEFAULT_INITSIZE to ALLOCSET_DEFAULT_MAXSIZE. Even if you
> allocate a good bit more than 8MB, having to additionally go through 20
> smaller chunks is going to be noticable until you reach a good bit higher
> number of blocks.

Well, let's try to help Johan get the information to us. I've attached
a quickly put together patch which adds some debug stuff to aset.c.
Johan, if you have a suitable test instance to try this on, can you
send us the filtered DEBUG output from the log messages starting with
"AllocSet" with and without your change? Just the output for just the
2nd execution of the query in question is fine. The first execution
is not useful as the cache of MemoryContexts may not be populated by
that time. It sounds like it's the foreign server that would need to
be patched with this to test it.

If you can send that in two files we should be able to easily see what
has changed in terms of malloc() calls between the two runs.

David