Re: Limiting memory allocation

On 5/17/22 18:30, Stephen Frost wrote:
> Greetings,
>
> On Tue, May 17, 2022 at 18:12 Tom Lane <tgl(at)sss(dot)pgh(dot)pa(dot)us
> <mailto:tgl(at)sss(dot)pgh(dot)pa(dot)us>> wrote:
>
> Jan Wieck <jan(at)wi3ck(dot)info <mailto:jan(at)wi3ck(dot)info>> writes:
> > On 5/17/22 15:42, Stephen Frost wrote:
> >> Thoughts?
>
> > Using cgroups one can actually force a certain process (or user, or
> > service) to use swap if and when that service is using more
> memory than
> > it was "expected" to use.
>
> I wonder if we shouldn't just provide documentation pointing to OS-level
> facilities like that one.  The kernel has a pretty trivial way to check
> the total memory used by a process.  We don't: it'd require tracking
> total
> space used in all our memory contexts, and then extracting some
> number out
> of our rear ends for allocations made directly from malloc.  In short,
> anything we do here will be slow and unreliable, unless you want to
> depend
> on platform-specific things like looking at /proc/self/maps.
>
>
> This isn’t actually a solution though and that’s the problem- you end up
> using swap but if you use more than “expected” the OOM killer comes in
> and happily blows you up anyway. Cgroups are containers and exactly what
> kube is doing.

Maybe I'm missing something, but what is it that you would actually
consider a solution? Knowing your current memory consumption doesn't
make the need for allocating some right now go away. What do you
envision the response of PostgreSQL to be if we had that information
about resource pressure? I don't see us using mallopt(3) or
malloc_trim(3) anywhere in the code, so I don't think any of our
processes give back unused heap at this point (please correct me if I'm
wrong). This means that even if we knew about the memory pressure of the
system, adjusting things like work_mem on the fly may not do much at
all, unless there is a constant turnover of backends.

So what do you propose PostgreSQL's response to high memory pressure to be?

Regards, Jan