Re: Connections closing due to "terminating connection due to administrator command"

On 07/27/2015 07:45 AM, Herouth Maoz wrote:
>
>> On 27 ביולי 2015, at 16:39, Adrian Klaver <adrian(dot)klaver(at)aklaver(dot)com> wrote:
>>>
>>> * Given that I did not terminate any backend connection interactively,
>>> why did I get a "terminating connection due to administrator
>>> command” message? Is there any situation where this message is
>>> issued without the administrator being involved?
>>
>> This error message comes from intervention by a program external to Postgres:
>>
>> http://www.postgresql.org/message-id/4564.1284559661@sss.pgh.pa.us
>>
>> So what OS and version are you running?
>
> OK, that’s a miss on my part, sorry: The OS is Debian GNU/Linux 7.
>
>>
>> What does the Java app do besides accessing the database?
>
> Well, it’s a message-passing application that basically up all the time, processing information sent in by our clients and distributing it. It is not running on the same machine as the database, it is not using a privileged database account (the user is the database’s owner, but not a super-user), and it’s not even running very complex queries. If more detailed information is needed, I’ll provide it as needed, as a full description of the system is going to be very lengthy, and I doubt it will be helpful, given that it’s not really capable of sending signals over to the database process, being on a separate machine.

Well what we are talking about is system users not database users, as
the error is an indication that some other program then Postgres is
shutting down the Postgres server.

>
>>
>> Do you have other programs that monitor/control the database?
>
> We have a backup script that runs at 4:02AM every day. Other than that, we just monitor the java program, so if the database fails, it shows up in that program’s log.
>
>>
>>
>>> * What could have caused the shutdown on the 21st of July? Again, I
>>> did not issue a shutdown request for that server, and the machine
>>> didn’t restart.
>>
>> Is there anything in logs before the above that might give a clue?
>
> No, that’s the thing. It’s the first entry in the log for days - the previous log entry is from July 8th.

Not sure what yo have set up for logging, but you might to crank it up.
13 days between entries for a system that is in use all the time seems
sort of light to me.

>
> NEW INFORMATION: I believe that today’s mishap is related to the July 21 shutdown problem, based on the logs of our Java program and an inquiry into the way it handles messages that go through a particular path. It seems that one of the modules has a bug and retains stale connections, and since messages that arrive through that path are rare, the first time one came after the July 21st shutdown was today.

So what is the module doing?

Or more the point what is the application doing to deal with the stale
connections?

>
> So I’m left with the question of what caused the shutdown on July 21st.
>
> Thank you,
> Herouth
>

--
Adrian Klaver
adrian(dot)klaver(at)aklaver(dot)com