Re: Limiting memory allocation

Le mercredi 18 mai 2022, 16:23:34 CEST Jan Wieck a écrit :
> On 5/17/22 18:30, Stephen Frost wrote:
> > Greetings,
> >
> > On Tue, May 17, 2022 at 18:12 Tom Lane <tgl(at)sss(dot)pgh(dot)pa(dot)us
> >
> > <mailto:tgl(at)sss(dot)pgh(dot)pa(dot)us>> wrote:
> > Jan Wieck <jan(at)wi3ck(dot)info <mailto:jan(at)wi3ck(dot)info>> writes:
> > > On 5/17/22 15:42, Stephen Frost wrote:
> > >> Thoughts?
> > >
> > > Using cgroups one can actually force a certain process (or user, or
> > > service) to use swap if and when that service is using more
> >
> > memory than
> >
> > > it was "expected" to use.
> >
> > I wonder if we shouldn't just provide documentation pointing to
> > OS-level
> > facilities like that one. The kernel has a pretty trivial way to
> > check
> > the total memory used by a process. We don't: it'd require tracking
> > total
> > space used in all our memory contexts, and then extracting some
> > number out
> > of our rear ends for allocations made directly from malloc. In short,
> > anything we do here will be slow and unreliable, unless you want to
> > depend
> > on platform-specific things like looking at /proc/self/maps.
> >
> > This isn’t actually a solution though and that’s the problem- you end up
> > using swap but if you use more than “expected” the OOM killer comes in
> > and happily blows you up anyway. Cgroups are containers and exactly what
> > kube is doing.
>
> Maybe I'm missing something, but what is it that you would actually
> consider a solution? Knowing your current memory consumption doesn't
> make the need for allocating some right now go away. What do you
> envision the response of PostgreSQL to be if we had that information
> about resource pressure? I don't see us using mallopt(3) or
> malloc_trim(3) anywhere in the code, so I don't think any of our
> processes give back unused heap at this point (please correct me if I'm
> wrong). This means that even if we knew about the memory pressure of the
> system, adjusting things like work_mem on the fly may not do much at
> all, unless there is a constant turnover of backends.

I'm not sure I understand your point: when we free() a pointer, malloc is
allowed to release the corresponding memory to the kernel. In the case of
glibc, it doesn't necessarily do so, but it trims the top of the heap if it is
in excess of M_TRIM_THRESHOLD. In the default glibc configuration, this
parameter is dynamically adjusted by mmap itself, to a maximum value of 64MB
IIRC. So any memory freed on the top of the heap totalling more than that
threshold ends up actually freed.

In another thread, I proposed to take control over this tuning instead of
letting malloc do it itself, as we may have better knowledge of the memory
allocations pattern than what malloc empirically discovers: in particular, we
could lower work_mem, adjust the threshold and maybe even call malloc_trim
ourselves when work_mem is lowered, to reduce the padding we may keep.

>
> So what do you propose PostgreSQL's response to high memory pressure to be?
>
>
> Regards, Jan

--
Ronan Dunklau