Re: Kubernetes, cgroups v2 and OOM killer - how to avoid?

On Sat, 2025-04-05 at 13:53 +0200, Ancoron Luciferis wrote:
> I've been investigating this topic every now and then but to this day
> have not come to a setup that consistently leads to a PostgreSQL backend
> process receiving an allocation error instead of being killed externally
> by the OOM killer.
>
> Why this is a problem for me? Because while applications are accessing
> their DBs (multiple services having their own DBs, some high-frequency),
> the whole server goes into recovery and kills all backends/connections.

You don't have to explain why that is a problem. It clearly is!

> Ideally, I'd find a configuration that only terminates one backend but
> leaves the others working.

There isn't, but what you really want is:

> I am wondering whether there is any way to receive a real ENOMEM inside
> a cgroup as soon as I try to allocate beyond its memory.max, instead of
> relying on the OOM killer.
>
> I know the recommendation is to have vm.overcommit_memory set to 2, but
> then that affects all workloads on the host, including critical infra
> like the kubelet, CNI, CSI, monitoring, ...

I cannot answer your question, but I'd like to make two suggestions:

1. set the PostgreSQL configuration parameters "work_mem", "shared_buffers",
"maintenance_work_mem" and "max_connections" low enough that you don't
go out of memory. A crash is bad, but a query failing with an "out of
memory" error isn't nice either.

2. If you want to run PostgreSQL seriously in Kubernetes, put all PostgreSQL
pods on a dedicated host machine where you can disable memory overcommit.

Yours,
Laurenz Albe