Constantly increasing per connection memory usage on pg 9.2

Dear Community,

we've recently moved one of our production databases from a 8.3 based
server to a 9.2 installation. Our usage patterns remained the same,
however, we noticed that the swap space in the server started to
decrease with time.

We have roughly 16 connections to this database in an almost constant
basis, pooled by a java connection pooler on the app side. This
connections never expire (nor did they do with the 8.3 db).
However, as you can see on this memory usage chart
http://i.imgur.com/2sRRj1J.png it shows a rolleccoaster like pattern.
When we restart our application, the connections are closed and the
memory is freed.

Our work_mem is 2MB, so it does not warrant the large memory allocation.
Also I am completely sure that this is the pg connection process that's
eating up the memory.

I've been digging the net for info for days, but this is the first time
I haven't found anything event remotely resembling our scenario. The
exact version number is: psql (EnterpriseDB) 9.2.1.3

Does anybody have any ideas what causes this?

regards
Eliott