27

I'm very new to network administration, so please regard that I'm not that experienced yet.

I have a Ubuntu root server with plesk panel.

Yesterday my friends and I noticed that the quality of speech on our TS3 got very bad. I sent some pings to the server and there was a very high packet loss. After that i googled a bit and found out that there is a auth.log. I downloaded it and scrolled a bit around, then I found this:

May 13 10:01:27 rs204941 sshd[9351]: input_userauth_request: invalid user student [preauth]
May 13 10:01:27 rs204941 sshd[9351]: pam_unix(sshd:auth): check pass; user unknown
May 13 10:01:27 rs204941 sshd[9351]: pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=112.220.198.102 
May 13 10:01:29 rs204941 sshd[9351]: Failed password for invalid user student from 112.220.198.102 port 39806 ssh2
May 13 10:01:29 rs204941 sshd[9351]: Received disconnect from 112.220.198.102: 11: Bye Bye [preauth]
May 13 10:01:31 rs204941 sshd[9353]: Invalid user student from 112.220.198.102

It seems like someone tried to log in over SSH many times. I scrolled a bit around, and saw, that this someone tries to use many different usernames: student, tech, psi, news,...

Hundreds of these logins were displayed in the file.

I looked up the traffic statistics at the website of my datacenter. It was only at 17MB per hour. I have a 100Mbit Backbone, so the data transfer itself seems not to be the problem.

At the moment i cannot get acces to the server in any way.

My Question is: how can i get acces again, how can i surpress this attack and prevent following attacks?

  • Still learning my stuff in this department.... But wouldn't this be a good use for port knocking? – mjrider May 13 '14 at 11:54
  • See also [Securing SSH server against bruteforcing](http://serverfault.com/q/174951/4276) and [Preventing brute force attacks against ssh?](http://serverfault.com/q/4188/4276) – Cristian Ciupitu May 13 '14 at 13:52
  • Add [How to secure ubuntu server from bruteforce ssh attacks?](http://askubuntu.com/q/32246/74792) and [how to prevent brute force attack on ssh?](http://superuser.com/q/491636/2357) to the list. – Cristian Ciupitu May 13 '14 at 13:58

5 Answers5

43

How to gain access?

It's not clear why you can't access your account.

If your machine is under attack or high load, you should talk to your provider about restricting access (IP Restrictions) or taking the server offline (disconnect from the Internet).

You might also require out of band access which your provider may be able to help with.

If somebody has compromised your server you may need to restore from backups or use a recovery image.

How to prevent attacks on your server, in particular SSH

best way to prevent brute force logons?

Don't let them get to your machine in the first place! There are plenty of ways to stop brute force attempts before they get to your host, or even at the SSH level.

Having said that, protecting your Operating System with something like fail2ban is a great idea. http://en.wikipedia.org/wiki/Fail2ban

Fail2ban is similar to DenyHosts ... but unlike DenyHosts which focuses on SSH, fail2ban can be configured to monitor any service that writes login attempts to a log file, and instead of using /etc/hosts.deny only to block IP addresses/hosts, fail2ban can use Netfilter/iptables and TCP Wrappers /etc/hosts.deny.

There are a number of important security techniques you should consider to help prevent brute force logins:

SSH:

  • Don't allow root to login
  • Don't allow ssh passwords (use private key authentication)
  • Don't listen on every interface
  • Create a network interface for SSH (e.g eth1), which is different to the interface you serve requests from (e.g eth0)
  • Don't use common usernames
  • Use an allow list, and only allow users that require SSH Access
  • If you require Internet Access...Restrict Access to a finite set of IPs. One static IP is ideal, however locking it down to x.x.0.0/16 is better than 0.0.0.0/0
  • If possible find a way to connect without Internet Access, that way you can deny all internet traffic for SSH (e.g with AWS you can get a direct connection that bypasses the Internet, it's called Direct Connect)
  • Use software like fail2ban to catch any brute force attacks
  • Make sure OS is always up to date, in particular security and ssh packages

Application:

  • Make sure your application is always up to date, in particular security packages
  • Lock down your application 'admin' pages. Many of the advice above applies to the admin area of your application too.
  • Password Protect your admin area, something like htpasswd for web console will project any underlying application vulnerabilities and create an extra barrier to entry
  • Lock down file permissions. 'Upload folders' are notorious for being entry points of all sorts of nasty stuff.
  • Consider putting your application behind a private network, and only exposing your front-end load balancer and a jumpbox (this is a typical setup in AWS using VPCs)
Drew Khoury
  • 4,569
  • 8
  • 26
  • 28
  • 1
    I have installed Fail2ban via https://help.ubuntu.com/community/Fail2ban It seems to be very powerfull and user-friendly(Mail notifications,...) –  May 13 '14 at 12:05
  • I got a message from my datacenter: There was a floodattack(what is this?) and this made the connection very bad for many servers. It was not a attack on my server especially. –  May 13 '14 at 12:11
  • See http://en.wikipedia.org/wiki/UDP_flood_attack – Drew Khoury May 13 '14 at 13:49
  • 1
    Interesting. So it seems that my datacenter was the target of the attack. But to know I got bruteforced besides this, is very good too. And Fail2ban is really a good tool. I will read about Iptables and SSH Config the next days and try to make it much more safe. –  May 13 '14 at 14:07
  • apt-get install fail2ban. It is the _second_ thing I apt install after: apt-get install git-core etckeeper – Dan Garthwaite May 13 '14 at 14:41
  • 3
    Configure openssh to listen to a port other than 22, and configure iptables to drop anything coming to port 22. If you don't expect to ssh in from any other country than your homeland, create a block list at https://www.countryipblocks.net/country_selection.php and use it with iptables. – ThoriumBR Apr 10 '15 at 17:58
  • Thanks @ThoriumBR for the country list. Even though I didn't go as far as blocking an entire country, I found the range my attacks were coming from in Hong Kong and dropped it. I have NEVER seen my logs so clean! Thanks for the pro tip. Sorry to whatever small region of Hong Kong that just went plonk - root out your spammer. :-) `$IPTABLES -A INPUT -i $EXTIF -s 43.255.188.0/22 -j drop-and-log-it` – moodboom Apr 16 '15 at 22:16
  • I've installed fail2ban but still getting failed login to my ssh account. Any suggestion? – Ray Coder Dec 07 '20 at 13:30
3

how can i surpress this attack and prevent following attacks

Usually i change the default ssh port from 22 to another like 1122. This prevent many automatic attacks from bot, but a simple port scan can detect it. Anyway:

vi /etc/ssh/sshd_config

and edit Port 22 to Port 1122, but this is not enough.

Automatic IPTables rules on bruteforce

i use log2iptables https://github.com/theMiddleBlue/log2iptables instead Fail2ban, because is a simple Bash script that parse any logfile with a regular expression and execute iptables. For example when 5 matches occur, log2iptables drop the specific ip address. It's cool because use Telegram API and can send me a message on my phone when he find a problem :)

hope this will help!

theMiddle
  • 64
  • 3
  • 10
    I'm guessing from the correlation between the project URL and your SF handle, that you're involved with this project. Please see [our guidelines](http://serverfault.com/help/behavior) on referencing your own products, in particular noting the requirement that "**you *must* disclose your affiliation in your answers**". – MadHatter Nov 19 '15 at 08:49
3

I've just put this together, run every 15 mins as a cronjob etc:

for z in `grep Invalid /var/log/auth.log | awk '{ print $NF }' | sort | uniq`
do
  count1=`grep $z /etc/hosts.deny | wc -l`
  count2=`grep Invalid /var/log/auth.log | grep $z | wc -l`
  if [ $count1 -eq 0 -a $count2 -gt 10 ] ; then
    current=`egrep "^ssh" /etc/hosts.deny | sed 's/sshd[ :,]*//'`
    sudo cp /etc/hosts.deny.bak /etc/hosts.deny
    sudo chmod 666 /etc/hosts.deny
    if [ $current ] ; then
      echo "sshd : $current , $z" >> /etc/hosts.deny
    else
      echo "sshd : $z" >> /etc/hosts.deny
    fi
    sudo chmod 644 /etc/hosts.deny
  fi
done
chicks
  • 3,639
  • 10
  • 26
  • 36
Mark
  • 31
  • 1
1

Automated Solution for Centos/RHEL to Block Bad Actors

Here's a script for Centos to check ssh failed logins for both invalid user accounts and bad passwords for valid accounts. If the source IP has hit us more than 3 times and is not already on the deny list, it gets added to the deny list. I run this every 15 minutes from root's crontab. I've also disallowed root logins via ssh, so the combination keeps things fairly quiet.

     #/bin/bash
     # Save a copy of the existing hosts.deny file for safety
     cp /etc/hosts.deny /etc/hosts.deny.bak
     # Get a list of the offending IP addresses and process them
     for z in `grep "Invalid\|Failed" /var/log/secure | awk '{ print $NF }' | sort | uniq`
     do
     # Get the number of times this IP hit us
     hits=`grep "Invalid\|Failed" /var/log/secure* | grep $z | wc -l`
     # Check whether this IP is already blocked
     blocked=`grep $z /etc/hosts.deny | wc -l`
     # If they hit us more than 3 times and are not already on the deny list
     # add them to the deny list
     if [ $hits -gt 3 -a $blocked -eq 0 ]
     then
          echo "sshd : $z" >> /etc/hosts.deny
     fi
     done
David Lash
  • 11
  • 1
0

This is my alternative solution for SSH attacks. The idea is keep closing SSH daemon if not use. No open port no attack. You can try it. It is open source https://github.com/indy99/nnet_port_guard

indy99
  • 1
  • 1
    Welcome to Server Fault! Your answer suggests a workable solution to the question is available via another website. The Stack Exchange family of Q&A websites generally frowns on this type of answer because other websites may move, get deleted, or changed. Please read [How do I write a good answer?](http://serverfault.com/help/how-to-answer) and consider revising your answer to include the steps required to resolve the issue. And don't forget to take the [site tour](http://serverfault.com/tour). – Paul Apr 10 '15 at 16:57