Is it normal to get hundreds of break-in attempts per day?

Question

I just checked my server's /var/log/auth.log and found that I'm getting over 500 failed password/break-in attempt notifications per day! My site is small, and its URL is obscure. Is this normal? Should I be taking any measures?

Until we locked down all unnecessary external ports, I remember not only did we get lots of hack attempts, but one day it was so bad that we were being hacked from two different countries -- at the same time! So yes, 100s of break-in attempts is perfectly normal. — Django Reinhardt, Mar 08 '11 at 14:53
We have servers that experience a new attack "sequence" once every 16 seconds. A single sequence is usually a batch of around 100 attempts across various ports. Just for kicks one day I turned on an unpatched server outside our firewall; it tooks less than 10 minutes from the time it was powered on for it to get pwnd. Point is the internet truly is a jungle; try not to get eaten. — NotMe, Mar 08 '11 at 18:55
My favorite is to tail the security log and see the user id combinations they try to use, hoping that you leave a guest id with an open password available. I had an ftp server port open that I used to communicate with work. I now shut it down and no longer expose the ports because of all the attempted break in traffic, heh. Most of the traffic was coming from China. I was shocked at first to see all of these attempts too. — jmq, Mar 09 '11 at 07:16
I can see I posted my question to the wrong site: http://superuser.com/questions/200896/windows-security-log-and-audit-failures — Justin C, Mar 09 '11 at 22:01
while I agree with others this is normal on common ports required (80, 443) I practically eliminated these attempts against my SSH port by simply changing the default port from 22 to something obscure like 6022 for example. Just doing that, alone, nearly eliminated 99% of that type of attack. — Kilo, Mar 11 '11 at 20:32
If you're going to change your SSH port, there are security reasons to keep it below port 1024 (only root can open ports < 1024, so it protects you from other users hijacking SSH). — Brendan Long, Mar 14 '11 at 17:54
This post really open my eyes. I took a look at my log and couldnt believe the attempts that were being made. Wow, I thought I was the only one trying to log into my server. — user489041, Oct 27 '11 at 19:33

score 207 · Accepted Answer · edited May 22 '12 at 20:26

207

In today's internet this is quite normal sadly. There are hordes of botnets trying to login to each server they find in whole IP networks. Typically, they use simple dictionary attacks on well-known accounts (like root or certain applications accounts).

The attack targets are not found via Google or DNS entries, but the attackers just try every IP address in a certain subnet (e.g. of known root-server hosting companies). So it doesn't matter that your URL (hence the DNS entry) is rather obscure.

That's why it is so important to:

disallow root-login in SSH (howto)
use strong passwords everywhere (also in your web applications)
for SSH, use public-key authentication if possible and disable password-auth completely (howto)

Additionally, you can install fail2ban which will scan the authlog and if it finds a certain amount of failed login attempts from an IP, it will proceed to add that IP to /etc/hosts.deny or iptables/netfilter in order to lock out the attacker for a few minutes.

In addition to the SSH attacks, it is also becoming common to scan your webserver for vulnerable web-applications (some blogging apps, CMSs, phpmyadmin, etc.). So make sure to keep those up-to-date and securely configured too!

edited May 22 '12 at 20:26

John Albietz

33
7

answered Mar 08 '11 at 11:35

Holger Just

3,315
1
16
23

21

Applications such as fail2ban can help a lot to 'temporarily' stop those bots from hitting your server at silly times in the morning :-) I have mine set up to ban 3 incorrect attempts for 24hours. – emtunc Mar 08 '11 at 13:06
46

And move ssh's port from 22 to 222. That works quite well. – Tom O'Connor Mar 08 '11 at 13:47
40

+1, public-key authentication only :) – 0xC0000022L Mar 08 '11 at 14:09
@Holger Just: to my knowledge fail2ban does not create iptables rules, but uses `/etc/hosts.deny` ... iptables rules would be difficult in fact, because order plays a big role in them, depending on the chain policy. So would it have to be added to the end, inserted at the front? ... – 0xC0000022L Mar 08 '11 at 14:12
@STATUS_ACCESS_DENIED: it depends on how you configure fail2ban. Out of the box for ssh creates it's own iptables chain (Chain fail2ban-SSH) and adds addresses to that. – user9517 Mar 08 '11 at 14:32
@Iain: yes, I noticed (looked it up quickly, while the hosts.deny was from memory :)). I edited the answer to include both. Thanks anyway for your comment. – 0xC0000022L Mar 08 '11 at 14:35
@STATUS_ACCESS_DENIED: adding to hosts.deny can be problematic too, depending on what's already there and what's in hosts.allow. The iptables solution doesn't require on-the-fly munging of a text file, either. It's better and more robust all around. – mattdm Mar 08 '11 at 17:09
@mattdm: of course a separate chain will provide more freedom to the tool, but I'm still skeptical about how it copes with rules outside its realm that aim at SSH as well. Any pointers? – 0xC0000022L Mar 08 '11 at 17:55
3

@STATUS_ACCESS_DENIED: the actions fail2ban takes are just lists of shell commands to run. So it's really flexible and easy to make work properly with any custom config. The best reference is to download it and look at `action.d/iptables.conf`. – mattdm Mar 08 '11 at 18:01
IMHO fail2ban is a flawed approach. Parsing arbitrary log data to trigger shell commands with root privileges means a major security risk in my book. – al. Mar 09 '11 at 11:18
@al: If you don't like that, you can also use the method described by STAZIS_ACCESS_DENIED below to use an iptables tarpit. But be aware that this might bring your system to a grinding halt when you face a DOS as the connection tables are quickly filled. – Holger Just Mar 09 '11 at 13:15
2

I suggest [DenyHosts](http://denyhosts.sourceforge.net/) It shares servers' intrusion attempt logs with a central server. This allows you to be "proactive" in your blocking of hosts by telling DenyHosts you want to block shared hosts that have a record of 6 or more failed attempts in the last week or so. It'll automatically clear out blocked hosts after a given time period and clean out it's iptables rules accordingly. – sholsinger Mar 09 '11 at 17:33
4

Blocking attackers like this is a waste of time. If you disable root login, there's a good chance that no one will ever even guess your correct login name, let alone password. SSH itself is already rate limiting password requests, so even if they know your user name (random bots won't), if you have a decent password, they'll never guess it. – Brendan Long Mar 09 '11 at 19:44
2

Instead of iptables rules or hosts.deny try using ipsets. You make a iptables rule to block every IP/network in the set, then add IPs to the set. The ipset implementation is very fast: much faster than the iptables rule chains. – Zan Lynx Mar 09 '11 at 20:21
1

My parent's server got hacked, probably through phpmyadmin - and started serving a spoofed bank site. Their ISP was kind enough to tell them they needed to fix it. whoops. – Wayne Werner Mar 10 '11 at 00:55
@Tom as long as you're sure you can connect out from non-standard ports. – David G Mar 11 '11 at 17:18
@david 443 is another good choice, actually, assuming you don't want to run SSL too. – Tom O'Connor Mar 11 '11 at 17:30
1

@Tom, security through obscurity is never a good option. Anyone with a port scanner like nmap will find your SSH server regardless of what port it's running on. `nmap -O -sV -p- (some host)` will scan every port, discover the service running on it and probe the OS type. – Registered User Mar 12 '11 at 21:26
@Deleted Account It's not STO. I have other services (denyhosts) running on that box too. Moving the port from the default does however, cut down on the chancers. I also detect port-scan attempts. Anyone who's portscanning me is obviously a little more determined. Luckily, i've got tools to detect them too. *evil laugh* – Tom O'Connor Mar 13 '11 at 19:35

score 58 · Answer 2 · answered Mar 08 '11 at 11:36

58

A few 100 is just fine... Last month I found one of my servers had 40k failed attempts. I went trough the trouble of plotting them: Map

Once I changed the ssh port and implemented Port Knocking, the number dropped to 0 :-)

answered Mar 08 '11 at 11:36

Bart De Vos

17,761
6
62
81

2

Nice map. I would love to know how to do this! – jftuga Mar 08 '11 at 15:27
9

@jftuga I got all the IP's from the logs first. `grep 'Failed password' /var/log/secure* | grep sshd | grep -o '[0-9]\{1,3\}\.[0-9]\{1,3\}\.[0-9]\{1,3\}\.[0-9]\{1,3\}' | sort | uniq`(remove the | uniq at the end if you want to allow duplicates). You can then put them in a CSV and upload it to zeemaps.com. I've seen better maps then mine, where they would use the count to color the map (green to red for the number of attempts per county) but I haven't jet figured that one out – Bart De Vos Mar 08 '11 at 15:50
3

What do you mean by 'implemented Port Knocking'? Is there an app i can install via apt-get to do this? The number dropping to 0 sounds nice – Mar 08 '11 at 21:46
1

@acidzombie24 I don't think its in the repositories, but you can download it here: http://www.zeroflux.org/projects/knock . It will allow you to close the port for SSH in iptables. If you then send a number of 'knocks' (aka TCP SYN-requests) to the right ports, it will open the port to ssh for just a brief moment, so you can get in. It's security true obscurity, which isn't the best, but still a nice add-on. – Bart De Vos Mar 09 '11 at 08:38
1

It looks like everybody in the UK is behaving or at the very least we're completely free bots over here... – MrEdmundo Mar 09 '11 at 11:51
18

Security through obscurity gets a bad wrap. It's perfectly fine as long it's _part_ of the overall strategy rather than the entire strategy. After all, what else is a password other than an obscure string? – Joel Coel Mar 09 '11 at 17:05
5

@Joel Coel, it's a _secret_ string, as opposed to most security through obscurity issues - an obscure, but not necessarily secret process. – tobyodavies Mar 10 '11 at 06:06
1

@tobyodavies: Well, a non-standard SSH port can be considered a secret integer, no? Granted, with 65535 ports available, it's a really weak secret (and most people will choose something like 222 or 220, which is not random at all), but it could be a useful filter to keep the simplest bots from bothering you; OTOH, I like my SSH on 22, so I can block all access (not just SSH) from a rogue host, via denyhosts. – Piskvor left the building Mar 11 '11 at 12:25
1

Changing the port is indeed very usefull. But try to keep it below 1024. You need elevated rights to run it on ports below 1024, this can come in handy when there is a breach and people may try to set-up a fake SSH-server. – Bart De Vos Aug 04 '11 at 08:05

score 29 · Answer 3 · edited Mar 09 '11 at 15:29

I for one use a "tarpit" in addition to only allowing public key authentication and disallowing root logins.

In netfilter there is a recent module, which you can use with (INPUT chain):

iptables -A INPUT -i if0 -p tcp --dport 22 -m state --state NEW -m recent --set --name tarpit --rsource
iptables -A INPUT -i if0 -p tcp --dport 22 -m state --state NEW -m recent --update --seconds 180 --hitcount 6 --name tarpit --rsource -j DROP
iptables -A INPUT -i if0 -p tcp --dport 22 -j ACCEPT

What that does is, that every attempt to connect to port 22 is listed by the recent module with IP and some other stuff under the name "tarpit" (if you're curious, look at /proc/net/xt_recent/tarpit). Obviously you can use other names.

To list or delist IPs use:

echo "+123.123.123.123" > /proc/net/xt_recent/tarpit
echo "-123.123.123.123" > /proc/net/xt_recent/tarpit

This rate-limits the attempts to 5 in 300 seconds. Please note that users with an existing connection are not bothered by that limit, because they already have an established connection and are allowed to create more (even above the rate limit).

Adjust the rules to your liking but make sure that they be added in that order (i.e. when adding then use them in this order, when inserting then in the reverse order).

This cuts down the noise immensely. It also provides actual security (against brute-forcing) unlike the perceived security of changing the port. However, I'd still recommend changing the port if it's feasible in your environment. It will cut down the noise level a lot as well ...

You can still combine this with fail2ban, although I've been running just fine without it and only the above rules.

EDIT:

It is possible to lock your self out doing this, so you can add something like the following that lets you clear you ban by knocking on a particular port:

iptables -A INPUT -i if0 -p tcp --dport <knockport> -m state --state NEW -m recent --name tarpit --remove

I use this and manage to block myself occasionally, so like to set another port up that you can "knock" on to clear your ban. — benlumley, Mar 09 '11 at 11:46
@benlumley: good point. Port knocking can be similarly useful to changing the default port - or even both of them in combination ... — 0xC0000022L, Mar 09 '11 at 12:52
@benlumley: saw your comment (now removed by Sam). I absolutely don't mind if the answer gets edited/improved ;) — 0xC0000022L, Mar 09 '11 at 17:36

score 15 · Answer 4 · edited Mar 08 '11 at 19:08

15

You could implement fail2ban, or similar methods like locking SSH to your IP. Sadly bots attempt to bruteforce access all the time so it is quite normal, you need to make sure you have a good password.

edited Mar 08 '11 at 19:08

Joe

1,765
15
23

answered Mar 08 '11 at 11:32

Jacob

9,114
4
44
56

3

If you're using SSH, consider public key authentication. This is a bit more secure than password authentication. – Piskvor left the building Mar 11 '11 at 12:39

score 12 · Answer 5 · edited Mar 10 '11 at 04:48

12

Yes. It's quite normal nowadays.

Please use only public key authentication for administrative purposes if possible. Generate a private key on you workstation:

$ ssh-keygen -t dsa

Copypaste the contents of ~/.ssh/id_dsa.pub to you servers ~/.ssh/authorized_keys (and /root/.ssh/authorized_keys, should you require direct root login).

Configure your servers /etc/ssh/sshd_config to only accept public key authentication:

PubkeyAuthentication yes
PasswordAuthentication no
PermitRootLogin without-password

If you have too many servers, you can use Puppet to run public keys and configurations to them.

Look into Denyhosts and fail2ban to block repeated SSH login attempts and see Snort if you require full IDS/IPS.

edited Mar 10 '11 at 04:48

Scott Pack

14,717
10
51
83

answered Mar 09 '11 at 23:32

jkj

592
4
12

2

I wouldn't recommend using public key authentication in SSH for shell access to your server. If your workstation is compromised or, even worse stolen, then someone now has open access to your servers without the need for a password. Public key authentication is more for situations where you need something like a script or program to be able to have SSH access to another system without needing a password so you don't have to embed plain-text passwords in your script/program. – Registered User Mar 12 '11 at 21:34
5

@Deleted Account: You can set a passphrase for the SSH private key. – Phil Cohen Mar 15 '11 at 22:57
2

"Registered User's" comment is misguided. Just to clarify: Always set a good password on your private key, and don't store the private key on any servers. Keep the private key on your own workstation. Once you add the key to an ssh-agent program, and enter your password, you can log into every system that has the public key installed, without once having to re-enter your password. Enable agent-forwarding in your ssh client so you can log in from server to server. Getting your private key stolen is bad, but with a decent password on it it's not as bad as a stolen password. – Martijn Heemels May 24 '11 at 20:47
Right, don't even think to store admin's private keys unencrypted. – yrk Apr 10 '12 at 09:02

score 8 · Answer 6 · answered Mar 09 '11 at 09:37

8

use http://denyhosts.sourceforge.net/

and yes, you should use public-key authentication and disable password auth.

answered Mar 09 '11 at 09:37

number5

266
3
12

Disabling password auth is not always a viable solution. – Publiccert Mar 11 '11 at 19:39

score 6 · Answer 7 · answered Mar 08 '11 at 11:33

The attempts are mechanized, so the numbers seem OK (yes they are high compared to some sites and the low compared to others). You should take the steps you normally have to: You consider your sites as attack targets every day, even when you do not detect an attack; not detecting an attack, does not mean it does not exist.

James Schek · Answer 8 · 2011-05-03T04:12:19.930

I'd say only getting only 500 is kind of low.

At a previous employer, one of the computer security researchers called the constant stream of break-in attempts the "internet equivalent of cosmic noise". He described it as a normal, continuous flow of malicious traffic that was searching for systems on the internet and automatically exploit scripts to attempt to hijack the system. Bot-nets and other malicious systems would perpetually scan and rescan the internet for vulnerable systems much like SETI.

score 6 · Answer 9 · edited Jun 11 '20 at 10:02

Yes,

this is common, but that doesn't mean you shouldn't fight the good fight. Here are some steps on how you can make your server more secure.

Avoid DNS associated IP addresses

You can greatly reduce this number in shared or colocation environments by disabling SSH access on any IP addresses associated with domain names. Unlisted non-domain IP addresses will receive less of this type of traffic, so buy an unlisted IP and only use this IP for SSH access.

Use a VPN for all SSH access

If you are in an environment in which you can implement IPsec/VPN to a private network within your server environment, this is ideal. Disable all SSH Internet access, make sure you have an integrated lights out solution. Setup your VPN, and only allow SSH access from your VPN.

Implement IP address rules for SSH access

If the VLAN isn't an option configure your router, or firewall rules to only allow SSH connections from a known IP address range.

If you follow these steps you will sleep much easier in the night knowing someone would have to compromise your hosting companies network to gain access to the server through SSH.

score 5 · Answer 10 · answered Mar 08 '11 at 11:43

Pretty normal to see hundreds of failed SSH connections.

If you have the option, I simply change my SSH port to something non-standard. It doesn't necessarily make your server any more secure, but it sure cleans up the logs (and lets you see anyone deliberately trying to break in!)

JRW · Answer 11 · 2011-03-08T18:27:25.963

5

In addition to using an automated lockout mechanism like fail2ban you have one more option: actually contact the abuse address ISP of the attacker. It may seem completely futile but in the case of the script-kiddie their ISP is more than willing to take action on them.

To find the abuse address, start with arin.net and lookup the IP address using whois. You may be redirected to another regional registry but eventually you can find the responsible ISP for the IP block which contains the address. Look for the abuse@ address or just mail the technical contact.

Send them a polite message with the relevant log file entries (make sure to remove any private information) and ask them to take action against the offending host.

edited Mar 08 '11 at 18:27

answered Mar 08 '11 at 18:20

JRW

159
2

4

We used to do this. However the amount of time spent versus the benefit received was so tiny it doesn't matter. – NotMe Mar 08 '11 at 18:50
1

A variant of this tactic which is more effective, but much riskier, is to report the ISP to the intermediate node. But you **MUST** have your report evidence solid. I got a whole ISP in grave trouble once doing this, because they were ignoring their abuse reports. – staticsan Mar 10 '11 at 00:10
1

One time I did this, the abuse message reached the hacker instead of someone in charge of the servers. Since then, I don't bother anymore, just too much trouble. – wump Mar 11 '11 at 15:23
This really isn't going to help in most cases and can be time consuming – RichVel Jun 21 '12 at 10:16

score 4 · Answer 12 · answered Mar 09 '11 at 10:34

I would recommend not using fail2ban but running SSH (and others) on a non-standard port. I don't believe in security by obscurity but I think that this is an excellent way to reduce the noise in your logs.

Failed logins on non-standard ports will be few and far between and can also indicate more targeted attacks.

You could even go a step further an install a SSH honeypot such as Kippo to 'let in' the bruteforcers and see what they would do given the chance.

Haha, Kippo looks very nice. I'm going to install it on a server just to see what they are trying to do. — wump, Mar 11 '11 at 15:29

score 4 · Answer 13 · answered Mar 11 '11 at 16:51

Yes, it's normal. What I tell clients in your situation with small websites.

Always be prepared to be hacked.

Have a copy of your website on a dev server. This can be your Windows desktop using XAMPP which you can get for free.

ALWAYS make changes to your dev server then upload them to your live website. If it is a CMS like Wordpress, make your posts on the dev server then copy and paste them into the live server.

NEVER download anything from your live website to your dev server.

Monitor your webpages regularly for any changes that you did not do. Specifically, hidden links to drugs or 'enhancement' products. You can find lots of browser add ins and programs that will do this for you.

If you are compromised. Notify your host, delete everything, change all passwords and upload your clean dev server to the now empty webserver. Work with your host to prevent a recurrence.

You should not need a security team for a small site. That is what your host is supposed to provide. If they do not, then get another host which is much easier to do when you have a dev server rather than trying to move the live server.

Hope this helps.

+1 for "Always be prepared to be hacked." – user78940 May 24 '11 at 23:05 — user78940, May 24 '11 at 23:05

score 3 · Answer 14 · answered Mar 08 '11 at 12:27

Another way of stopping it (as I personally don't like to move the SSH port): decide if you are able to list all networks from which you'll ever want to login, then only allow these to access your SSH port.

WHOIS entries of local ISPs helped me to reduce attacks to 1-2 login attempts a month (back then it was about 1k/day). I detected those by still using denyhosts.

score 3 · Answer 15 · edited Mar 10 '11 at 00:37

In addition to the other excellent suggestions you've already received, I also like to use the AllowUsers directive if appropriate for the given server. This allows only specified users to login via SSH which greatly reduces the possibility of gaining access through an insecurely configured guest/service/system account.

Example:

AllowUsers admin jsmith jdoe

The option AllowUsers specifies and controls which users can access ssh services. Multiple users can be specified, separated by spaces.

score 3 · Answer 16 · answered Mar 12 '11 at 01:28

Yes, it is normal. You can :

Reduce the opportunity for attack by using fwknop

Fwknop is the one of the better port knock implementations because it is not spoofable and actually authenticates as opposed to just authorizing a connection.

You can change the port that Openssh uses but you are not really improving security.
Fortify ssh authentication using Google-authenticator or wikid

This will protect password based attacks and the possibility of a determined attacker / targeted attack compromising your admin machine and stealing your ssh-key & password combo.

Just look at the latest pwn2own comp to see how easy it is for a skilled attacker to compromise your fully patched admin box.

score 1 · Answer 17 · answered Mar 08 '11 at 11:36

Sadly this is quite normal. You should consider adding something like fail2ban to your system to automatically detect and ban the attackers. If you don't already do so you should also consider only using ssh with public keys and don't allow root login over ssh. If use ftp to transfer files to the system, consider using scp/sftp instead.

score 1 · Answer 18 · answered Mar 08 '11 at 15:54

I implemented port knocking, and have a few probes per day. They don't get a connection, so they go away. I log and report all access to the ports involved.

I have also run fail2ban with Shorewall as a firewall to temporarily blacklist persistent attackers.

If you don't need Internet access to SSH disable it. If you have a few known addresses which need remote access, limit access to those addresses.

Limiting access to authorized-keys can also be helpful.

Christoffer Hammarström · Answer 19 · 2011-03-09T13:16:32.763

0

I use pam_abl to temporarily blacklist brute forcers, and it works great. I think it feels better to have authorization in PAM using its own database rather than depend on hosts.deny or iptables.

Another plus is that pam_abl does not depend on scanning log files.

edited Mar 09 '11 at 13:16

answered Mar 09 '11 at 13:06

Christoffer Hammarström

111
7

score 0 · Answer 20 · edited Mar 11 '11 at 09:02

0

It's completely normal these days.
You can set up "burst" limit on firewall for incoming new connections on the SSH port,
or install one of many log parsers a'la fail2ban or change the SSH port ;).

The last one is the easiest one. On heavy loaded machines such break-in attempts can heavy really bad influence to whole system.

--
Regards,
Robert

edited Mar 11 '11 at 09:02

Kyle

105
2

answered Mar 09 '11 at 22:56

Robert

346
1
7

score 0 · Answer 21 · answered Mar 11 '11 at 17:06

Yes it's normal.

I just changed the ssh port away from the standard 22. My server, my rules :) just edit /etc/ssh/sshd_config, change the port and restart the service. The only down side is that you must remember to add that port to configuration to every ssh client you use.

score 0 · Answer 22 · answered Aug 08 '12 at 04:24

Disable root login (In every linux system root user exist so bots can easily guess the user name).After logging in as normal user you can switch to root either by su or sudo.
change default port from 22
Allow ssh access from known ip's only
Use a strong alpha-numeric-password for the user with ssh access

Is it normal to get hundreds of break-in attempts per day?

22 Answers22

Yes,

Avoid DNS associated IP addresses

Use a VPN for all SSH access

Implement IP address rules for SSH access

Linked

Related