Re: Postgres dying after many failed logins

On Mon, 8 Nov 2021 at 18:15, Lynn Carol Johnson <lcj34(at)cornell(dot)edu> wrote:

> Hello all-
>
> I have a postgres instance running on an AWS ec2 machine (not RDS ). It
> is receiving many hits from the hacker address 209.141.53.139. Because
> this address has been implicated in hacker attempts previously, I have the
> pg_hba.conf set to explicitly reject this address ( so I can see how many
> times it hits). https://www.abuseipdb.com/check/209.141.53.139
>
> Note there are other restrictions on which addresses are allowed to
> connect, and we have non-default passwords setup on this db.
>
> I'm finding that after postgres is hit by and rejects many connections, it
> dies. I haven't been able to find any documentation that explains failed
> connections causing the server to die but that is what I'm seeing. In the
> log file there are multiple of these messages:
>
> 2021-11-04 15:14:46.537 UTC [1513486] postgres(at)postgres FATAL:
> pg_hba.conf rejects connection for host "209.141.53.139", user "postgres",
> database "postgres", SSL on
> 2021-11-04 15:14:46.709 UTC [1513488] postgres(at)postgres FATAL:
> pg_hba.conf rejects connection for host "209.141.53.139", user "postgres",
> database "postgres", SSL off
> 2021-11-04 15:14:48.566 UTC [1513494] postgres(at)postgres FATAL:
> pg_hba.conf rejects connection for host "209.141.53.139", user "postgres",
> database "postgres", SSL on
> 2021-11-04 15:14:48.761 UTC [1513505] postgres(at)postgres FATAL:
> pg_hba.conf rejects connection for host "209.141.53.139", user "postgres",
> database "postgres", SSL off
> ....
> 2021-11-05 11:13:49.519 UTC [1834715] postgres(at)postgres FATAL:
> pg_hba.conf rejects connection for host "209.141.53.139", user "postgres",
> database "postgres", SSL on
> 2021-11-05 11:13:49.702 UTC [1834718] postgres(at)postgres FATAL:
> pg_hba.conf rejects connection for host "209.141.53.139", user "postgres",
> database "postgres", SSL off
> 2021-11-05 14:35:09.197 UTC [1451469] LOG: received smart shutdown request
> 2021-11-05 14:35:09.222 UTC [1451660] postgres(at)breedbase FATAL:
> terminating connection due to administrator command
> 2021-11-05 14:35:09.222 UTC [1451662] postgres(at)breedbase FATAL:
> terminating connection due to administrator command
>
> And after the time span seen here, the log shows a smart shutdown request
> message shown above. All connections are terminated and the system is shut
> down.
>
>
> My question: Is this expected behavior, ie that the server will shutdown
> after hours of failed attempts? Is there anything I can do to prevent
> this, or further secure the database? The hackers are unsuccessful due to
> the rejected connections, but it is a problem that the database server is
> continually shut down.
>
>
I am not sure pg_hba can handle that attack imho.
you need to have something at the network layer or proxy layer to handle
bot attack kind of requests.

i think all cloud providers have ddos protection of some kind like Shield
(AWS) etc.
We used akamai for ddos mitigation, and used various rules to tarpit, block
ips etc, user agent filtering, location etc depending on the type of attack.
manual management of ips may or may not work. We have seen cases where the
moment you reject an ip, they bots learn and start attacking from a new ip
etc.

i think having a proxy layer via envoyproxy/haproxy/nginx etc for simple
ddos protection would work fine.
but you would need network layer protections as well some ddos mitigation
service
for your apps.

Application-Layer DDoS Attack Protection with HAProxy - HAProxy Technologies
<https://www.haproxy.com/blog/application-layer-ddos-attack-protection-with-haproxy/>

Also as a general practice, the database is not to be exposed to the public.

AWS Shield - Amazon Web Services (AWS) <https://aws.amazon.com/shield/>

just my opinion. i have doubts postgresql database hba alone can handle
ddos.