kswapd 100%, swap full, vm.swappiness=0

Hardware:
48 core AMD Magny Cours (4x12)
128G 1333MHz memory
34 15k6 drives, 2 hot spares, rest in RAID-1 pairs, 1 set for OS, 4
for pg_xlog, rest for /data/base
LSI 8888 RAID controller
OS:
Ubuntu 10.04

uname -a
Linux bigassdbserver 2.6.32-24-generic #38-Ubuntu SMP Mon Jul 5
09:20:59 UTC 2010 x86_64 GNU/Linux

scheduler = noop for all drive sets.
Settings for sysctl.conf:
vm.zone_reclaim_mode = 0
kernel.shmmax = 33554432000
kernel.shmall = 2097152000
kernel.shmmni = 4096
vm.swappiness = 0
vm.dirty_ratio = 2
vm.dirty_background_ratio = 1

$ free
total used free shared buffers cached
Mem: 131651412 104986524 26664888 0 910804 91170764
-/+ buffers/cache: 12904956 118746456
Swap: 0 0 0

(swap is now off with sudo swapoff -a, it fixed the problem)

It's twin, the read slave, looks like this:

$ free
total used free shared buffers cached
Mem: 131651412 110364700 21286712 0 702144 96771656
-/+ buffers/cache: 12890900 118760512
Swap: 25388024 940 25387084

So, this morning, the machine goes into 100% swap usage. four kswapds
are running at 100% CPU in mostly D state. Load climbs to 300.
Server gets a little slow. Swapoff -a fixes it.

This makes no sense to me. The machine had 90G+ in kernel cache, and
was NOT running out of memory in any way. Swappiness is 0.

Any advice on this, reporting it to the kernel guys etc welcome.

--
To understand recursion, one must first understand recursion.