Why is this system swapping?

Hello,

I am trying to understand what I need to do for this system to stop
using swap. Maybe it's something simple, or obvious for the situation.
I'd appreciate some thoughts/suggestions.

Some background:

This is a quad XEON (yes, Dell) with 12GB of RAM, pg 7.4...pretty heavy
on concurrent usage. With peak traffic (db allows 1000 connections, in
line with the number of app servers and connection pools for each)
following is from 'top' (sorted by mem) Shared_buffers is 170MB,
sort_mem 2MB. Both WAL and pgdata are on separate LUNs on fibre channel
storage, RAID10.

972 processes: 971 sleeping, 1 running, 0 zombie, 0 stopped

CPU states: cpu user nice system irq softirq iowait idle

total 57.2% 0.0% 23.2% 0.0% 3.6% 82.8% 232.4%

cpu00 22.0% 0.0% 9.1% 0.1% 0.9% 18.7% 48.8%

cpu01 17.5% 0.0% 5.8% 0.0% 2.3% 19.7% 54.4%

cpu02 7.8% 0.0% 3.7% 0.0% 0.0% 20.8% 67.5%

cpu03 9.7% 0.0% 4.4% 0.0% 0.5% 23.6% 61.5%

Mem: 12081744k av, 12055220k used, 26524k free, 0k shrd,
71828k buff

9020480k actv, 1741348k in_d, 237396k in_c

Swap: 4096532k av, 472872k used, 3623660k free 9911176k
cached

PID USER PRI NI SIZE RSS SHARE STAT %CPU %MEM TIME CPU
COMMAND

21397 postgres 22 0 181M 180M 175M D 25.9 1.5 85:17 0
postmaster

23820 postgres 15 0 178M 177M 175M S 0.0 1.5 1:53 3
postmaster

24428 postgres 15 0 178M 177M 175M S 0.0 1.5 1:35 3
postmaster

24392 postgres 15 0 178M 177M 175M S 2.7 1.5 2:07 2
postmaster

23610 postgres 15 0 178M 177M 175M S 0.0 1.5 0:29 2
postmaster

24395 postgres 15 0 178M 177M 175M S 0.0 1.5 1:12 1
postmaster

...

...

-bash-2.05b$ free

total used free shared
buffers cached

Mem: 12081744 12055536 26208 0 66704
9943988

-/+ buffers/cache: 2044844 10036900

Swap: 4096532 512744 3583788

As you can see the system starts utilizing swap at some point, with so
many processes. Some time ago we had decided to keep the connections
from the pool open for longer periods of time, possibly to avoid
connection maintenance overhead on the db. At that time the traffic was
not as high as it is today, which might be causing this, because for the
most part, non-idle postmaster processes are only a few, except when the
system becomes busy and suddenly you see a lot of selects piling up, and
load averages shooting upwards. I am thinking closing out connections
sooner might help the system release some memory to the kernel. Swapping
adds up to the IO, although OS is on separate channel than postgres.

I can add more memory, but I want to make sure I haven't missed out
something obvious.

Thanks!

Anjan

************************************************************************
******************
This e-mail and any files transmitted with it are intended for the use
of the
addressee(s) only and may be confidential and covered by the
attorney/client
and other privileges. If you received this e-mail in error, please
notify the
sender; do not disclose, copy, distribute, or take any action in
reliance on
the contents of this information; and delete it from your system. Any
other
use of this e-mail is prohibited.
************************************************************************
******************