Connection terminated by the server causes deadlock in jdbc client side connection

From: Leonardo Frittelli <leonardo(dot)frittelli(at)gmail(dot)com>
To: pgsql-jdbc(at)postgresql(dot)org
Subject: Connection terminated by the server causes deadlock in jdbc client side connection
Date: 2015-10-07 16:33:19
Message-ID: CACh06N3O40kLx9YLH6cT8_DTEdRVt+G+tCBAinrjzhtECHfgUQ@mail.gmail.com
Views: Raw Message | Whole Thread | Download mbox | Resend email
Thread:
Lists: pgsql-jdbc

Hi,

We are experiencing very frequent deadlocks in pgsql jdbc connections. The
scenario is a replicated database with hot_standby = on

At times of high volumes of queries in both the primary and the replicated
server, we sometimes get the following log indicating that the server has
terminated the connection in the replicated database.
FATAL: terminating connection due to conflict with recovery
DETAIL: User query might have needed to see row versions that must be
removed.
HINT: In a moment you should be able to reconnect to the database and
repeat your command.

This is expected and we see no issue with that. What I did not expect,
however, is that in the JDBC client side, the connection is deadlocked.

java.lang.Thread.State: RUNNABLE
at java.net.SocketOutputStream.socketWrite0(Native Method)
at java.net.SocketOutputStream.socketWrite(Unknown Source)
at java.net.SocketOutputStream.write(Unknown Source)
at java.io.BufferedOutputStream.flushBuffer(Unknown Source)
at java.io.BufferedOutputStream.flush(Unknown Source)
- locked <0x0000000700742e30> (a java.io.BufferedOutputStream)
at org.postgresql.core.PGStream.flush(PGStream.java:518)
at
org.postgresql.core.v3.ProtocolConnectionImpl.close(ProtocolConnectionImpl.java:136)
at
org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:280)
- locked <0x0000000700742fd0> (a org.postgresql.core.v3.QueryExecutorImpl)
at
org.postgresql.jdbc2.AbstractJdbc2Statement.execute(AbstractJdbc2Statement.java:547)
at
org.postgresql.jdbc2.AbstractJdbc2Statement.executeWithFlags(AbstractJdbc2Statement.java:417)
at
org.postgresql.jdbc2.AbstractJdbc2Statement.executeQuery(AbstractJdbc2Statement.java:302)
...

Looking at the Postgres JDBC code, I notice that
ProtocolConnectionImpl.close() (invoked by the exception handler in
QueryExecutorImpl.execute) is trying to 'gracefully close' by sending an
'X' to the server before actually closing the socket.

...
if (logger.logDebug()) logger.debug(" FE=> Terminate");
pgStream.SendChar('X'); pgStream.flush(); pgStream.close(); ...

Does this make sense in a scenario of a connection which has already been
terminated by the server side?

At times of high load, all connections in the pool get eventually locked
with exactly the same stack trace.

I'd appreciate any advice on how to handle this. Could this be a bug in the
JDBC driver?

Thanks,

Leonardo

Responses

Browse pgsql-jdbc by date

  From Date Subject
Next Message Dave Cramer 2015-10-07 18:23:37 Re: Connection terminated by the server causes deadlock in jdbc client side connection
Previous Message Christopher BROWN 2015-10-07 12:42:33 Re: JDBC-94: "Multiple resultsets were returned by query" in query end with "; "