Re: Changes to Linux OOM killer in 2.6.36

On Thu, Nov 18, 2010 at 19:43, Greg Smith <greg(at)2ndquadrant(dot)com> wrote:
> Last month's new Linux kernel 2.6.36 includes a rewrite of the out of memory
> killer:
> http://lwn.net/Articles/391222/
> http://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6.git;a=commitdiff;h=a63d83f427fbce97a6cea0db2e64b0eb8435cd10

Yeah, Ive been following this somewhat closely...
Also of interest is the recent thread about reverting the new oom
(don't know if it will happen, but maybe they won't deprecate
oom_adj):
http://lkml.org/lkml/2010/11/14/5

> The new "badness" method totals the task's RSS and swap as a percentage of
> RAM, where the old one scored starting with the total memory used by the
> process.  I *think* that this is an improvement for PostgreSQL, based on the
> sort of data I see with:

Well, it seems to be an improvement. If I look at the oom_score on a
2.6.36 box ruining postgres I get:
$ cd /proc; for a in [0-9]*; do echo `cat $a/oom_score` $a `perl
-pes/'\0.*$'// < $a/cmdline`; done|grep -v ^0|sort -n |less
1 1309 supervising syslog-ng
1 1310 /usr/sbin/syslog-ng
1 1336 /usr/sbin/crond
1 1368 /usr/sbin/irqbalance
1 1485 /usr/sbin/ntpd
1 1495 /usr/local/bin/pgbouncer
1 1506 /sbin/agetty
....
1 3391 /var/lib/postgres/pgsql-9.0/bin/postgres
1 3393 postgres: writer process
1 3394 postgres: wal writer process
1 3395 postgres: autovacuum launcher process
1 3396 postgres: stats collector process
1 4110 postgres: joshua wopr [local] idle
2 4109 postgres: joshua wopr [local] idle

So in this case it should kill one of the backends *before* the
postmaster. Ignoring that backend... it looks like postmaster has the
same score as every other process on the system. It also has a has a
higher RSS than most, so I suspect it will still get killed first:
$ ps ax -o rss,pid,size,vsize,args | sort -n
...
2416 1680 588 46548 /usr/lib/postfix/master
2424 1696 640 46748 qmgr -l -t fifo -u
2956 3395 2416 244644 postgres: autovacuum launcher process
3116 2216 720 65464 sshd: alex [priv]
4096 3393 1088 243316 postgres: writer process
6592 4110 2516 246808 postgres: joshua wopr [local] idle
11756 3391 900 243128 /var/lib/postgres/pgsql-9.0/bin/postgres
32640 4109 9084 255564 postgres: joshua wopr [local] idle in transaction

So I think we will still need to protect the postmaster from OOM :(.

> One thing that's definitely changed is the interface used to control turning
> off the OOM killer.

Grr... Whatever happens to a stable userspace abi?

> I don't think it's worth doing anything to the database code until tests on
> the newer kernel confirm whether this whole thing is even necessary anymore.

+1