Re: Properly handle OOM death?

On Mar 13, 2023, at 11:10 AM, Joe Conway <mail(at)joeconway(dot)com> wrote:
>
> On 3/13/23 14:50, Israel Brewster wrote:
>> Looks like V2:
>> root(at)novarupta:~# stat -fc %T /sys/fs/cgroup/
>> cgroup2fs
>
> Interesting -- it does indeed look like you are using cgroup v2
>
> So the file you want to look at in that case is:
> 8<-----------
> cat /sys/fs/cgroup/system.slice/system-postgresql.slice/postgresql(at)14(dot)service/memory.max
> 4294967296
>
> cat /sys/fs/cgroup/system.slice/system-postgresql.slice/postgresql(at)14(dot)service/memory.high
> 3221225472
> 8<-----------
> If the value comes back as "max" it means no limit is set.

This does, in fact, appear to be the case here:

root(at)novarupta:~# cat /sys/fs/cgroup/system.slice/system-postgresql.slice/postgresql(at)13-main(dot)service/memory.max
max
root(at)novarupta:~# cat /sys/fs/cgroup/system.slice/system-postgresql.slice/postgresql(at)13-main(dot)service/memory.high
max
root(at)novarupta:~#

which would presumably indicate that it’s a system level limit being exceeded, rather than a postgresql specific one? The syslog specifically says "Memory cgroup out of memory”, if that means something (this is my first exposure to cgroups, if you couldn’t tell).
---
Israel Brewster
Software Engineer
Alaska Volcano Observatory
Geophysical Institute - UAF
2156 Koyukuk Drive
Fairbanks AK 99775-7320
Work: 907-474-5172
cell: 907-328-9145

>
> In this example (on my Linux Mint machine with a custom systemd unit file) I have memory.max set to 4G and memory.high set to 3G.
>
> The value of memory.max determines when the OOM killer will strike. The value of memory.high will determine when the kernel goes into aggressive memory reclaim (trying to avoid memory.max and thus an OOM kill).
>
> The corresponding/relevant systemd unit file parameters are:
> 8<-----------
> MemoryAccounting=yes
> MemoryHigh=3G
> MemoryMax=4G
> 8<-----------
>
> There are other ways that memory.max may get set, but it seems most likely that the systemd unit file is doing it (if it is in fact set).
>
> --
> Joe Conway
> PostgreSQL Contributors Team
> RDS Open Source Databases
> Amazon Web Services: https://aws.amazon.com
>