Re: Deadlock detection

Simon Riggs wrote:

> If not, other suggestions and viewpoints welcome. Thanks,

If you want a minimally invasive approach (in terms of rewriting driver
code) and some extra overhead isn't a problem then I would suggest
something like this:

(1) spawn a separate write thread for each connection; it reads from an
unbounded buffer

(2)instead of writes from the thread running a query going directly to
the real stream, they instead go into the buffer (without blocking), and
the corresponding write thread picks them up for immediate writing

(3) implement OutputStream.flush() to wait for the buffer to empty,
*with a timeout*. If you hit the timeout while waiting to flush, then
you've detected a potential deadlock case and can dump out some state
and then continue anyway.

The usual behaviour of the driver is (approximately): write some query
data, write a flush or sync message, flush the socket output stream,
start to read results. In the deadlock case, we run out of (socket)
write buffer space at some point before completing the stream flush, and
the socket buffer never empties because the server is blocked trying to
write to us.

With the above changes, in that case the write thread will block and the
internal Java-side write buffer will grow. Then OutputStream.flush()
will block and eventually timeout, and the deadlock should be broken.

The obvious costs of this are the extra thread per connection, the
context switches needed to actually do the writes, and however much
space you need for all the write buffers.

Turn OutputStream.flush() into a no-op and you actually have a fix for
the deadlock case entirely - it's just not a very cheap fix.

You may want to put an upper bound on the internal buffer size and throw
IOException if you hit it. I suppose that a connection error is better
than either a silent deadlock or exhausting your heap..

-O