Re: Properly handle OOM death?

> On Mar 13, 2023, at 10:37 AM, Joe Conway <mail(at)joeconway(dot)com> wrote:
>
> On 3/13/23 13:55, Israel Brewster wrote:
>> 1) They reference a “Memory cgroup out of memory”, which refers back
>> to the opening comment on Joe Conway’s message - this would imply to
>> me that I *AM* running with a cgroup memory.limit set. Not sure how
>> that changes things?
>
> cgroup memory limit is enforced regardless of the actual host level memory pressure. As an example, if your host VM has 128 GB of memory, but your cgroup memory limit is 512MB, you will get an OOM kill when the sum memory usage of all of your postgres processes (and anything else sharing the same cgroup) exceeds 512 MB, even if the host VM has nothing else going on consuming memory.
>
> You can check if a memory is set by reading the corresponding virtual file, e.g:
>
> 8<-------------------
> # cat /sys/fs/cgroup/memory/system.slice/postgresql.service/memory.limit_in_bytes
> 9223372036854710272
> 8<-------------------
>
> A few notes:
> 1/ The specific path to memory.limit_in_bytes might vary, but this example is the default for the RHEL 8 postgresql 10 RPM.

Not finding that file specifically (this is probably too much info, but…):

root(at)novarupta:~# ls /sys/fs/cgroup/system.slice/
-.mount cgroup.threads dev-hugepages.mount memory.events.local memory.swap.events proc-diskstats.mount ssh.service system-postgresql.slice systemd-resolved.service
accounts-daemon.service cgroup.type dev-lxc-console.mount memory.high memory.swap.high proc-loadavg.mount sys-devices-system-cpu-online.mount systemd-initctl.socket systemd-sysctl.service
cgroup.controllers console-getty.service dev-lxc-tty1.mount memory.low memory.swap.max proc-meminfo.mount sys-devices-virtual-net.mount systemd-journal-flush.service systemd-sysusers.service
cgroup.events console-setup.service dev-lxc-tty2.mount memory.max networkd-dispatcher.service proc-stat.mount sys-fs-fuse-connections.mount systemd-journald-audit.socket systemd-tmpfiles-setup-dev.service
cgroup.freeze cpu.pressure dev-mqueue.mount memory.min pids.current proc-swaps.mount sys-kernel-debug.mount systemd-journald-dev-log.socket systemd-tmpfiles-setup.service
cgroup.max.depth cpu.stat dev-ptmx.mount memory.numa_stat pids.events proc-sys-kernel-random-boot_id.mount syslog.socket systemd-journald.service systemd-update-utmp.service
cgroup.max.descendants cron.service io.pressure memory.oom.group pids.max proc-sys-net.mount sysstat.service systemd-journald.socket systemd-user-sessions.service
cgroup.procs data.mount keyboard-setup.service memory.pressure pool.mount 'proc-sysrq\x2dtrigger.mount' 'system-container\x2dgetty.slice' systemd-logind.service ufw.service
cgroup.stat dbus.service memory.current memory.stat postfix.service proc-uptime.mount system-modprobe.slice systemd-networkd.service uuidd.socket
cgroup.subtree_control dbus.socket memory.events memory.swap.current proc-cpuinfo.mount rsyslog.service system-postfix.slice systemd-remount-fs.service

root(at)novarupta:~# ls /sys/fs/cgroup/system.slice/system-postgresql.slice/
cgroup.controllers cgroup.max.depth cgroup.stat cgroup.type io.pressure memory.events.local memory.max memory.oom.group memory.swap.current memory.swap.max pids.max
cgroup.events cgroup.max.descendants cgroup.subtree_control cpu.pressure memory.current memory.high memory.min memory.pressure memory.swap.events pids.current postgresql(at)13-main(dot)service
cgroup.freeze cgroup.procs cgroup.threads cpu.stat memory.events memory.low memory.numa_stat memory.stat memory.swap.high pids.events

root(at)novarupta:~# ls /sys/fs/cgroup/system.slice/system-postgresql.slice/postgresql(at)13-main(dot)service/
cgroup.controllers cgroup.max.depth cgroup.stat cgroup.type io.pressure memory.events.local memory.max memory.oom.group memory.swap.current memory.swap.max pids.max
cgroup.events cgroup.max.descendants cgroup.subtree_control cpu.pressure memory.current memory.high memory.min memory.pressure memory.swap.events pids.current
cgroup.freeze cgroup.procs cgroup.threads cpu.stat memory.events memory.low memory.numa_stat memory.stat memory.swap.high pids.events

>
> 2/ The value above, 9223372036854710272 basically means "no limit" has been set.
>
> 3/ The example assumes cgroup v1. There are very few distro's that enable cgroup v2 by default, and generally I have not seen much cgroup v2 usage in the wild (although I strongly recommend it), but if you are using cgroup v2 the names have changed. You can check by doing:
>
> 8<--cgroupv2 enabled-----------------
> # stat -fc %T /sys/fs/cgroup/
> cgroup2fs
> 8<--cgroupv1 enabled-----------------
> # stat -fc %T /sys/fs/cgroup/
> tmpfs
> 8<-------------------

Looks like V2:

root(at)novarupta:~# stat -fc %T /sys/fs/cgroup/
cgroup2fs
root(at)novarupta:~# lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description: Ubuntu 20.04.3 LTS
Release: 20.04
Codename: focal

---
Israel Brewster
Software Engineer
Alaska Volcano Observatory
Geophysical Institute - UAF
2156 Koyukuk Drive
Fairbanks AK 99775-7320
Work: 907-474-5172
cell: 907-328-9145

>
>> 2) All the entries contain the line "oom_score_adj:0”, which would
>> seem to imply that the postmaster, with its -900 score is not being
>> directly targeted by the OOM killer.
>
> Sounds correct
>
> --
> Joe Conway
> PostgreSQL Contributors Team
> RDS Open Source Databases
> Amazon Web Services: https://aws.amazon.com
>