Re: Connection terminated by the server causes deadlock in jdbc client side connection

Stephen,

You are correct that this is not a concurrency deadlock.
I was referring to the fact that the TCP connection is indeed left 'half
open' with the jdbc client side still assuming it's 'properly' open and
thus trying to send a graceful close command over the socket ("X" in v2 or
"X4" in v3) . It is this action that indefinitely blocks the client thread,
which is then no longer usable by the application.

Dave's point is a good one too. If the server is not properly closing the
connection, this might be causing the lock in the client side.
By the way, our server is version 9.3. I don't know if this would reproduce
as is in 9.4.

All of this said, I still think it is conceptually incorrect to attempt a
graceful closure after an IO error in the client.

Regards,

Leonardo

On 8 October 2015 at 09:57, Dave Cramer <pg(at)fastcrypt(dot)com> wrote:

>
> On 8 October 2015 at 08:47, Steffen Heil (Mailinglisten) <
> lists(at)steffen-heil(dot)de> wrote:
>
>> Hi
>>
>>
>> Do have a deadlock, you need at least two threads.
>> Also they both need to be BLOCKED, not RUNNABLE.
>> socketWrite should throw an exception, if the connection is closed for
>> sending.
>>
>> So I assume the server closed the connection for transfer from the server
>> to the client, but the connection remained half-open from the client to the
>> server.
>> Then on the other hand, you write should have gone through...
>>
>> What happens if the server just discards the connection. In other words
> it doesn't close it. It just terminates without closing any open
> connections ?
>
>
> Dave
>
>
>>
>> What locks here and why?
>>
>>
>> Regards,
>> Steffen
>>
>>
>>
>> -----Ursprüngliche Nachricht-----
>> Von: pgsql-jdbc-owner(at)postgresql(dot)org [mailto:
>> pgsql-jdbc-owner(at)postgresql(dot)org] Im Auftrag von Leonardo Frittelli
>> Gesendet: Mittwoch, 7. Oktober 2015 18:33
>> An: pgsql-jdbc(at)postgresql(dot)org
>> Betreff: [JDBC] Connection terminated by the server causes deadlock in
>> jdbc client side connection
>>
>> Hi,
>>
>> We are experiencing very frequent deadlocks in pgsql jdbc connections.
>> The scenario is a replicated database with hot_standby = on
>>
>> At times of high volumes of queries in both the primary and the
>> replicated server, we sometimes get the following log indicating that the
>> server has terminated the connection in the replicated database.
>> FATAL: terminating connection due to conflict with recovery
>> DETAIL: User query might have needed to see row versions that must be
>> removed.
>> HINT: In a moment you should be able to reconnect to the database and
>> repeat your command.
>>
>> This is expected and we see no issue with that. What I did not expect,
>> however, is that in the JDBC client side, the connection is deadlocked.
>>
>> java.lang.Thread.State: RUNNABLE
>> at java.net.SocketOutputStream.socketWrite0(Native Method)
>> at java.net.SocketOutputStream.socketWrite(Unknown Source)
>> at java.net.SocketOutputStream.write(Unknown Source)
>> at java.io.BufferedOutputStream.flushBuffer(Unknown Source)
>> at java.io.BufferedOutputStream.flush(Unknown Source)
>> - locked <0x0000000700742e30> (a java.io.BufferedOutputStream)
>> at org.postgresql.core.PGStream.flush(PGStream.java:518)
>> at
>> org.postgresql.core.v3.ProtocolConnectionImpl.close(ProtocolConnectionImpl.java:136)
>> at
>> org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:280)
>> - locked <0x0000000700742fd0> (a org.postgresql.core.v3.QueryExecutorImpl)
>> at
>> org.postgresql.jdbc2.AbstractJdbc2Statement.execute(AbstractJdbc2Statement.java:547)
>> at
>> org.postgresql.jdbc2.AbstractJdbc2Statement.executeWithFlags(AbstractJdbc2Statement.java:417)
>> at
>> org.postgresql.jdbc2.AbstractJdbc2Statement.executeQuery(AbstractJdbc2Statement.java:302)
>> ...
>>
>>
>> Looking at the Postgres JDBC code, I notice that
>> ProtocolConnectionImpl.close() (invoked by the exception handler in
>> QueryExecutorImpl.execute) is trying to 'gracefully close' by sending an
>> 'X' to the server before actually closing the socket.
>>
>>
>> ...
>> if (logger.logDebug()) logger.debug(" FE=> Terminate");
>> pgStream.SendChar('X'); pgStream.flush(); pgStream.close(); ...
>>
>> Does this make sense in a scenario of a connection which has already been
>> terminated by the server side?
>>
>> At times of high load, all connections in the pool get eventually locked
>> with exactly the same stack trace.
>>
>> I'd appreciate any advice on how to handle this. Could this be a bug in
>> the JDBC driver?
>>
>> Thanks,
>>
>> Leonardo
>>
>
>