Re: CLOSE_WAIT pileup and Application Timeout

On Fri, Oct 4, 2024 at 9:17 PM Adrian Klaver <adrian(dot)klaver(at)aklaver(dot)com>
wrote:

> On 10/3/24 21:29, KK CHN wrote:
> > List,
> >
> > I am facing a network (TCP IP connection closing issue) .
> >
> > Running a mobile tablet application, Android application to update the
> > status of vehicles fleet say around 1000 numbers installed with the app
> > on each vehicle along with a vehicle tracking application server
> > solution based on Java and Wildfly with PosrgreSQL16 backend.
> >
>
> >
> > The running vehicles may disconnect or be unable to send the location
> > data in between if the mobile data coverage is less or absent in a
> > particular area where data coverage is nil or signal strength less.
> >
> > The server on which the backend application runs most often ( a week's
> > time or so) shows connection timeout and is unable to serve tracking
> > of the vehicles further.
> >
> > When we restart the Wildfly server the application returns to normal.
> > again the issue repeats after a week or two.
>
> Seems the issue is in the application server. What is not clear to me is
> whether the connection timeout you refer to is from the mobile devices
> to the application or the application to the Postgres server?

its from mobile devices to application server. When I do a restart of
application server everything backs to normal. But after a period of time
again it cripples. That time when I netstat on Application VM lots of
CLOSE_WAIT states as indicated.

> I'm
> guessing the latter as I would expect the mobile devices to drop
> connections more often then weekly.

> Yes mobile devices may drops connections at any point of time if it
> reaches an area where signal strength is poor( eg; Underground parking or
> near the areas where mobile data coverage is poor.
> >

The topology is mobile devices connect and update the location via
application VM then finally in PGSQL VM.

The application server and Database server both separate virtual
machines. Application server hangs most often not the database VM.
Since there are other applications which update to the database VM without
any issue. The DB VM caters all the writes from other applications. But
those applications are different, not fleet management one.

>
> > In the Server machine when this bottleneck occurs I am seeing a lot
> > of TCP/IP CLOSE_WAIT ( 3000 to 5000 ) when the server backend becomes
> > unresponsive.
>
> Again not clear, are you referring to the application or the Postgres
> database running on the server?
>
> >
> > What is the root cause of this issue ? Is it due to the android
> > application unable to send the CLOSE_WAIT ACK due to poor mobile data
> > connectivity ?
> >
> >
> > If so, how do people address this issue ? and what may be a fix ?
> >
> > Any directions / or reference material most welcome.
> >
> > Thank you,
> > Krishane
> >
> >
> >
> >
> >
>
> --
> Adrian Klaver
> adrian(dot)klaver(at)aklaver(dot)com
>
>