Re: Bug (and fix): leaks of TCP connections when connected

Marcus Andree S. Magalhaes wrote:
> <snip>
>
>>In theory the discarded connections should eventually be garbage
>>collected and closed, right? So at least the leak is bounded.
>
> I don't think so. We had a similar problem here.
>
> After many hours of debugging, we came to the fact that either
> java garbage collector does _not_ close open connections

It's not the GC that directly closes the open connections; it just
causes unreachable objects to be collected. It's the responsibility of
object that holds out-of-heap resources to do whatever cleanup (OS-level
close() calls etc) is needed when it becomes unreachable, traditionally
via finalize() (although you can also use ReferenceQueue). I'd expect
the socket implementation to do this.

> or do it
> after a long time (much longer than one would expect).

Typically finalization and Reference clearing happens only on full
collections (it's not guaranteed to happen this way, though), so I'd
expect the connections to be collected then. It can take some time for a
full GC to happen, depending on your heap settings. We see problems in
NIO buffer allocation when System.gc() is disabled (see below) on a
quiescent system that can have intervals of over a day between full GCs.

If you leak sockets or some other resource rapidly enough that you run
out of the resources at the OS level (or indirectly hit a resource
limitation on the remote side e.g. server connection limit) before the
owning objects are collected, you will have a problem.

NIO direct buffer allocation hits exactly this issue, and ends up
issuing a System.gc() when it thinks it's out of direct buffer space to
try to avoid the problem. There are some bugs in Sun's implementation
though which make it hard to use (it forces a stop-the-world GC which is
bad if you are using the CMS collector, and there is a race between the
allocating thread returning from System.gc() and buffer references being
cleared in the reference-handler thread). Assuming those problems can be
fixed, there's an argument for doing the same thing in the socket
allocation code (and anywhere else that allocates out-of-heap resources)
too. It becomes a question of how far to take it: if the server refuses
a connection with "too many connections", should you GC and try again if
that collected any connections to the server in question?

See http://bugs.sun.com/bugdatabase/view_bug.do?bug_id=5025281

(apologies for rambling :)

-O