18

I am being attacked right now from Tor nodes which are doing 404 requests to my HTTP server. It is from one IP but when i use the DROP iptables rule, it starts again from another IP in a matter of seconds.

It started to ask requests to the cgi-bin folder, but now it is doing requests in my images folder. So based on my investigation it is a tool connected to TOR looking for something and right now it drives me mad.

I tried fail2ban, mod_evasive and mod_security, but those programs get triggered when the other side is looking for one thing or banging at the door at one port. But this tool is looking at every request for a different file.

45.63.100.91 - - [26/Dec/2016:03:27:17 +0100] "HEAD //images/imgsupport.php HTTP/1.1" 404 - "-" "Mozilla/5.0 (Windows NT 10.0; WOW64; Trident/7.0; MATBJS; rv:11.0) like Gecko"
45.63.100.91 - - [26/Dec/2016:03:27:17 +0100] "HEAD //images/imgppexg.php HTTP/1.1" 404 - "-" "Mozilla/5.0 (Windows NT 10.0; WOW64; Trident/7.0; MATBJS; rv:11.0) like Gecko"
45.63.100.91 - - [26/Dec/2016:03:27:17 +0100] "HEAD //images/imgppantivirussoft.php HTTP/1.1" 404 - "-" "Mozilla/5.0 (Windows NT 10.0; WOW64; Trident/7.0; MATBJS; rv:11.0) like Gecko"
45.63.100.91 - - [26/Dec/2016:03:27:17 +0100] "HEAD //images/imgwin95.php HTTP/1.1" 404 - "-" "Mozilla/5.0 (Windows NT 10.0; WOW64; Trident/7.0; MATBJS; rv:11.0) like Gecko"
45.63.100.91 - - [26/Dec/2016:03:27:17 +0100] "HEAD //images/imgnws.php HTTP/1.1" 404 - "-" "Mozilla/5.0 (Windows NT 10.0; WOW64; Trident/7.0; MATBJS; rv:11.0) like Gecko"
45.63.100.91 - - [26/Dec/2016:03:27:17 +0100] "HEAD //images/imgroundcorner.php HTTP/1.1" 404 - "-" "Mozilla/5.0 (Windows NT 10.0; WOW64; Trident/7.0; MATBJS; rv:11.0) like Gecko"
45.63.100.91 - - [26/Dec/2016:03:27:17 +0100] "HEAD //images/imgppcdl.php HTTP/1.1" 404 - "-" "Mozilla/5.0 (Windows NT 10.0; WOW64; Trident/7.0; MATBJS; rv:11.0) like Gecko"
45.63.100.91 - - [26/Dec/2016:03:27:17 +0100] "HEAD //images/yellowbuy3.php HTTP/1.1" 404 - "-" "Mozilla/5.0 (Windows NT 10.0; WOW64; Trident/7.0; MATBJS; rv:11.0) like Gecko"
45.63.100.91 - - [26/Dec/2016:03:27:17 +0100] "HEAD //images/imgvirusinfo.php HTTP/1.1" 404 - "-" "Mozilla/5.0 (Windows NT 10.0; WOW64; Trident/7.0; MATBJS; rv:11.0) like Gecko"
45.63.100.91 - - [26/Dec/2016:03:27:17 +0100] "HEAD //images/escan4.php HTTP/1.1" 404 - "-" "Mozilla/5.0 (Windows NT 10.0; WOW64; Trident/7.0; MATBJS; rv:11.0) like Gecko"
45.63.100.91 - - [26/Dec/2016:03:27:17 +0100] "HEAD //images/imgredline.php HTTP/1.1" 404 - "-" "Mozilla/5.0 (Windows NT 10.0; WOW64; Trident/7.0; MATBJS; rv:11.0) like Gecko"
45.63.100.91 - - [26/Dec/2016:03:27:17 +0100] "HEAD //images/imgonlinescan.php HTTP/1.1" 404 - "-" "Mozilla/5.0 (Windows NT 10.0; WOW64; Trident/7.0; MATBJS; rv:11.0) like Gecko"
45.63.100.91 - - [26/Dec/2016:03:27:17 +0100] "HEAD //images/ram1.php HTTP/1.1" 404 - "-" "Mozilla/5.0 (Windows NT 10.0; WOW64; Trident/7.0; MATBJS; rv:11.0) like Gecko"
45.63.100.91 - - [26/Dec/2016:03:27:17 +0100] "HEAD //images/imgmanualscan.php HTTP/1.1" 404 - "-" "Mozilla/5.0 (Windows NT 10.0; WOW64; Trident/7.0; MATBJS; rv:11.0) like Gecko"
45.63.100.91 - - [26/Dec/2016:03:27:17 +0100] "HEAD //images/userdefine.php HTTP/1.1" 404 - "-" "Mozilla/5.0 (Windows NT 10.0; WOW64; Trident/7.0; MATBJS; rv:11.0) like Gecko"
45.63.100.91 - - [26/Dec/2016:03:27:17 +0100] "HEAD //images/betterinterface.php HTTP/1.1" 404 - "-" "Mozilla/5.0 (Windows NT 10.0; WOW64; Trident/7.0; MATBJS; rv:11.0) like Gecko"

You see it is looking in the folder images for a specific .php file, but at every request it is looking for a different php file. So, what the hell is this thing doing?

This also eats up my bandwidth, so I am really desperate what I need to do now. Anyone an idea how I can block this?

Server setup: CentOS 7 (OpenVZ, so I am stuck with a kernel where ipset isn't working) with Apache.

techraf
  • 9,141
  • 11
  • 44
  • 62
Alex
  • 181
  • 1
  • 6
  • Try upgrading ipset to 6.30 directly from netfilter.org. See if it is the kernel or ipset that is actually broken. – cybernard Dec 26 '16 at 05:03
  • Are you getting similar requests from other IP addresses? This looks suspiciously like botnet behavior – Henry F Dec 26 '16 at 05:43
  • Only when i block that IP...then it starts again, but where it was stopped, with another IP. All TOR ip's. – Alex Dec 26 '16 at 07:56
  • 4
    These are just HEAD requests - they should be some of the smallest http requests you can get. What kind of bandwidth use are you seeing to make you worry about overall bandwidth use? – Xiong Chiamiov Dec 26 '16 at 17:23
  • 4
    Also, if they're getting 404s, that doesn't really matter - it means they aren't successfully finding vulnerabilities. Any filtering you put in place has a decent chance to hurt legitimate traffic, so unless this is actually causing harm to your site, it's best to ignore it. – Xiong Chiamiov Dec 26 '16 at 17:32
  • 2
    You can certainly write a custom fail2ban match for this sort of thing. – Michael Hampton Dec 26 '16 at 19:08

5 Answers5

17

You could drop packets from tor nodes all together if you like with ip tables. List of tor nodes can be found at:

Ref 1: https://check.torproject.org/cgi-bin/TorBulkExitList.py
Ref 2: https://www.dan.me.uk/tornodes

Here is a bash one liner to block all traffic from tor to your web server. There are currently about 2000 tor nodes online now that support port 80. So you will be inserting quite a few iptable's wouldnt really call this a long term fix but should stop the attack. 

wget "https://check.torproject.org/cgi-bin/TorBulkExitList.py?ip=8.8.8.8&port=" -O /tmp/ip.lst && sed -i -e '1,3d' /tmp/ip.lst && for i in $(cat /tmp/ip.lst); do iptables -A INPUT -p tcp -s $i --match multiport --dports 80,443 -j DROP; done
Tim Jonas
  • 807
  • 1
  • 7
  • 19
6

You can drop packets that contain specific string and apperantely all requests contain: 'HEAD //images/' string.

I suggest the following rule for now and later you can remove it:

iptables -I INPUT -p tcp --dport 80 -m string --algo bm --string 'HEAD //images/' -j DROP

Opaida
  • 323
  • 1
  • 3
  • 1
    You could do this, but they just move to different folders, and you will still end up with 100's of rules. – cybernard Dec 26 '16 at 04:42
  • If this was the case, the user can block "HEAD " request in the same way and in 99% this will not affect the functionality of the website. – Opaida Dec 26 '16 at 05:17
  • Hi, i tried this, but it didn't do much. I even tried it with only HEAD, but it didn't stopped any request. – Alex Dec 26 '16 at 07:38
  • Make sure about the HTTP port.. the rule considers you are using port 80 (default). – Opaida Dec 26 '16 at 07:51
  • Is this: `iptables -I INPUT -p tcp --dport 443 -m string --algo bm --string 'HEAD //*' -j DROP` `iptables -I INPUT -p tcp --dport 80 -m string --algo bm --string 'HEAD //*' -j DROP` – Alex Dec 26 '16 at 08:52
  • ok? I don't have commenting permissions, so i need to type over again when i hit enter. The logs are looking ok now, but i don't know if that comes from the tor node list rule i gave in or from these rules. So hopefully someone can tell me if above rules are ok? – Alex Dec 26 '16 at 08:54
  • 6
    @Alex It looks OK from here but the best way to ensure this rule is well written is to disable the TOR-related rule you wrote before. Especially knowing that some people can legitimately be using TOR while using your website properly - you don't want to ban these ones. – Shlublu Dec 26 '16 at 08:56
  • @Alex no need for HTTPS rule (443 port) as traffic will be encrypted. – Opaida Dec 26 '16 at 09:02
  • @Alex you can check iptables statistics `iptables -nvL` – Opaida Dec 26 '16 at 09:04
  • Thank you, this is what i have now: http://pastebin.com/aDEqWqRE – Alex Dec 26 '16 at 09:18
  • As there is no match, then maybe you have GZIP/compression is enabled in the server. I suggest to capture one attack packet (using tcpdump) and extract one unique string from there then update the rules accordingly. – Opaida Dec 26 '16 at 09:24
  • @Shlublu you are right, but i don't have an option at the moment. They are using the TOR network to flood my server. I am trying to get the --string rule fixed, but atm i have no luck with it. – Alex Dec 26 '16 at 09:50
  • @Alex I see, and it's better to be up and running with less legitimate users than being down with no users at all, one can only agree. Good luck! – Shlublu Dec 26 '16 at 09:55
  • @Opaida when i do `tcpdump -i venet0 port 443` and try to connect to my server with Torbrowser i don't see any specific unique string in the output which i could use. Maybe i am doing it wrong :) – Alex Dec 26 '16 at 09:56
  • @Opaida you mean with GZIP/compression available that my log files are automatically compressed? Not that i know of...all my logs gets a new file after 7 days, but it won't get compressed. Just something like `access-log-20161211` – Alex Dec 26 '16 at 10:04
  • @Alex having that many rules is going to drive your CPU utilization up. You need ipset for efficiency. iptables -N web iptables -J web -p tcp -m tcp -m multiport --dports 80,443 Then move all your rules to the web chain. Then at least only a portion of the traffic will have to run the gaunlet. – cybernard Dec 26 '16 at 15:18
  • @cybernard, i know, but for now this is the only thing i can do right now. Load seems to be stable and eventually i will remove them again, because i want Tor users also can use my service. Just checking now how i can convert my OpenVZ box into a KVM box so i can use ipset in the future, but for now and 10 hours of hard work it is good enough to overcome the holidays – Alex Dec 26 '16 at 15:35
  • KVM is an installable program, you can install inside of OpenVZ box. **Run a full backup first** Is there some reason you don't attempt to upgrade the software of OpenVZ? https://openvz.org/028_to_042_kernel_upgrade – cybernard Dec 26 '16 at 16:00
  • Also read http://superuser.com/questions/649128/is-it-possible-and-safe-to-update-debian-inside-vps There is a comment about an issue with debian from Dec of 2014, which I am sure is fixed by now. – cybernard Dec 26 '16 at 16:03
3

My web server has this all the time, and after months of examining logs I can see there are 100's of different things they will eventually ask for.

One of thousands of bot networks is mapping your apache for vulnerable components. Hoping to get lucky, with a common component that it knows how to exploit. I get bots searching for phpmyadmin all the time. Along with dozens of other things.

Then of course if does get lucky, they will eventually deploy further bad things so they can use your computer for their own nefarious deeds.

Virtualize with something you can install a modern kernel, so you can use ipset. Maybe KVM, docker uses the built-in kernel so that is no good.

There botnet is likely 10's of thousands of computers so you won't be able to effectively block this, especially not without ipset.

Also as long as you are using TOR they have access to even more IP addresses from random people.

Option #2

Replace the HTTP_NOT_FOUND.html.var file found, at least on my system, /usr/share/apache2/error

You could create a HTTP_NOT_FOUND.html.var.php file then change the apache file,/etc/apache2/errors.conf, to point to said file.

Then the 1st attempt would be the last.

cybernard
  • 518
  • 2
  • 10
  • All i found out is that it is a scriptkiddie which found a script to search for something through the TOR network. I'm glad it is not a botnet :) – Alex Dec 26 '16 at 08:17
3

If your security model permits passing traffic to third party, then the easiest and most effective solution may be to front your application with Cloudflare. At the time of this writing, Cloudflare pretty much automatically blocks all Tor connection unless they solve a CAPTCHA page and they also have Web Application Firewall you can configure to further filter traffic before reaching your server so the attacker don't consume your server bandwidth.

Otherwise, you may have some luck by configuring fail2ban. fail2ban is a piece of software that puts temporary IP block by changing the OS firewall policy when someone fails authentication too many times. In your case, fail authentication would be someone that keeps 404-ing.

Lie Ryan
  • 31,089
  • 6
  • 68
  • 93
  • 4
    atm i can't use Cloudflare because i use a EV SSL. I need to have a business account from them and my business isn't that big yet to pay 200USD per month just for the website. – Alex Dec 26 '16 at 10:07
0

First, don't panic and make sure you have a towel with you!

From the information you have provided, I don't think you're being attacked, just scanned. These days, this is 'normal' behaviour and something you're likely to see on a regular basis. Think of it as someone casing your joint, looking for an open window or unlocked door. They are just looking to see if there is an easy way which they can use to gain an entry.

There are many scripts and tools out there, as botnets which just scan the internet looking for possibly vulnerable systems. Whether this is a prelude to something more serious is almost impossible to determine (yet). A lot will depend on whether your site looks valuable and to what extent your system is up-to-date.

  • Make sure your software is all patched and up-to-date
  • Make sure you have disabled any services running on your server which you don't need/use
  • Make sure your web server configuration is correct i.e. only has enabled features you use, has standard things, like a correctly configured robots.txt, does not have enabled any modules you don't need etc.

You mention this traffic is killing your bandwidth. This makes me suspect you're running the web server on a system with a low bandwidth as these requests are quite light weight, i.e. just HEAD requests. If this is the case, perhaps you need to re-think whether running the server is actually a good idea.

When your putting a server on the Internet, you're putting up something which is public - you're saying "Hey, I've got a server here, come and have a look". You need to be prepared for when someone will come, and you have no control over who does. Trying to block this sort of traffic will send you mad - if you let it, you will just end up in a game of 'wak-a-mole'. Don't even bother trying to block this level of traffic because as soon as you do, you will likely just get something vary similar from a different IP address. You will end up spinning your wheels writing filters and adjusting firewalls.

Setting up a server and putting it on the web is almost a trivial task these days. However administering such systems is not, and something you need to consider carefully. Putting it on a server at the end of a home or small business DSL link or similar is almost always a bad idea. It will likely have a hit on your bandwidth, will frequently take more time to administer than you expect and you're unlikely to have the underlying infrastructure necessary to provide a robust and reliable service.

My suggestion would be to consider hosting your web service with one of the many different hosting services out there. There is a large variety of choices and you can almost certainly find something which meets your needs from low level Infrastructure as a Services (IaaS), through to Platform as a Servide (PaaS) and Software as a Service (SaaS). Each service offers varying levels of flexibility and varying costs which you can select from. You will be able to find something with the right balance of cost and features with enough research.

If you don't want to pay to host your service, you really need to re-consider what you're doing. There are costs associated with all forms of hosting. The difference is how those costs are absorbed - either you pay with money or you pay with time and your own resources.

grochmal
  • 5,677
  • 2
  • 19
  • 30
Tim X
  • 3,242
  • 13
  • 13
  • First, i am not panicking ;) second, i have enough bandwidth (30TB per server), but that doesn't mean that someone just can eat it because they like it to conquer a server. I am just monitoring my server very closely. The reason why i wanted to stop this "attack" is because it is a mail service and they were trying to hack into it to abuse my service to spread spam. So, for me personally i wanted to stop this attack right away to let them know that it is worthless to try. I am a server admin with 20+ servers online, so i know what i am doing and i just wanted to stop this. Happy holidays. – Alex Dec 30 '16 at 22:23
  • Not sure what you mean by 30TB per server with respect to bandwidth - sounds like storage size rather than network bandwidth. Regardless, you would need to be seeing a LOT of these HEAD requests before you would see any noticeable impact on your bandwidth assuming a reasonable connection i.e. !Gbps or better. At any rate, this sort of scanning is vary frequent and while it is good to monitor, preventing it is near impossible. Assume you are always under 'attack'. A focus on hardening rather than blocking will generally give a more robust and effective defence. – Tim X Dec 31 '16 at 00:40
  • No it is bandwidth. Like i said, i know what i am doing here. I am a Red Hat server admin on a highschool for several years now and have for my private company already 20+ servers running in a DC. I know what bandwidth is and what hdd space is lol. That said, at the moment of the, according to you, innocent attack my server got abused by someone who made several accounts and was spamming around the 10000 mails per day while i have rules like one recipient per 30 seconds. If at that time also a scanning job is running, then i can only acknowledge that i am under attack and need to do something – Alex Jan 01 '17 at 01:50
  • I'm not questioning your knowledge or experience, I'm saying I don't understand what '30Tb per server' means with respect to bandwidth. I'm accustomed to bandwidth referring to amount of data transferred over an amount of time. For example, where I work we have 2 dual core 10 Gbit per second links. We have a class B network with a few hundred servers and about 4k desktops. Your metric has no reference to time, so I don't understand what it means wrt bandwidth. I do know researchers managed to achieve 1Tb per second over optic fibre for the 1st time in 2016. – Tim X Jan 01 '17 at 22:08