4

I run a server with apache2 on an ubuntu box. It is a crypto mining pool. Everything was running normal even with the large inflow of people accessing our site. Then out of nowhere our cpu usage across all 8 cores is at 98 to 100%

I use a reverse proxy in front of the server to help mitigate DDOS attacks but this is something I have never encountered before.

netstat -an | grep :80 | grep SYN | wc -l gives an output of 100+ while this is happening.

Killing apache immediately drops the load to normal levels and allows my clients to continue to use the other services on the server. But we have to have our front end working.

I have tried several iptables rules but they do not work. I have modified my sysctl according to some suggestions:

net.ipv4.conf.default.rp_filter = 1
net.ipv4.conf.all.rp_filter = 1
net.ipv4.tcp_syncookies = 1
net.ipv4.conf.all.accept_redirects = 0
net.ipv6.conf.all.accept_redirects = 0
net.ipv4.conf.all.send_redirects = 0
net.ipv4.conf.all.accept_source_route = 0
net.ipv6.conf.all.accept_source_route = 0
net.core.wmem_max = 16777216
net.core.rmem_max = 16777216
net.ipv4.tcp_rmem = 4096 87380 16777216
net.ipv4.tcp_wmem = 4096 65536 16777216
net.ipv4.tcp_mem = 128000 200000 262144
net.ipv4.tcp_syncookies = 1
net.ipv4.tcp_synack_retries = 1
net.ipv4.ip_local_port_range = 1024 65000
net.ipv4.tcp_max_syn_backlog = 8192
net.ipv4.tcp_max_tw_buckets = 1800000
net.ipv4.tcp_fin_timeout = 30
net.ipv4.tcp_tw_reuse = 1
net.ipv4.tcp_timestamps = 0
net.ipv4.tcp_tw_recycle = 1
vm.swappiness = 5
net.ipv4.tcp_syncookies = 1

but that does not seem to mitigate the issue.

I am at my wits end and I need to get the website up and functioning again.

ANY and all suggestions would be appreciated.

Thank you.

user3491412
  • 41
  • 2

2 Answers2

1

You can find the number of top 10 (or other number) current TCP connections per IP with netstat -tan| grep -v 'LISTEN'| awk '{print $5}'| grep -v 'and' |grep -v 'Address' |cut -d':' -f1 |sort -n | uniq -c | sort -rn | head -n10

And also the top 10 IPs in Apache log files with cd /var/log/apache2; for i in ./access.log*; do echo $i; cat $i | awk '{print $1}'| sort -n | uniq -c | sort -rn | head -n10; done

Once you determine the abusive IPs you can block them with iptables -A INPUT -s 1.2.3.4 -j DROP.

You can also rate-limit the connections with iptables, For example if you don't want more than 20 connections per minute from a source:

iptables -I INPUT -p tcp --dport 80 -m state --state NEW -m recent --set

iptables -I INPUT -p tcp --dport 80 -m state --state NEW -m recent --update --seconds 60 --hitcount 20 -j DROP

LinuxDevOps
  • 1,754
  • 9
  • 14
  • Do not use state tracking rules on webservers, even more so when said webservers are under attack. You'll quickly fill up your state tracking tables and effectively DDOS yourself. – devicenull Apr 03 '14 at 02:05
  • You may want to tighten up that command a bit. It seems silly to run `netstat -a` and then immediately follow that by `grep -v LISTEN`. Why not skip the netstat option that ements the listening services in the first place. Perhaps `netstat -tn | awk '/^tcp/ {gsub(":.*","",$5);print $5}' | sort -n| uniq -c | sort -rn` instead. – Zoredache Apr 03 '14 at 02:10
  • @Zoredache yes, command is highly unoptimized thanks for the improvement – LinuxDevOps Apr 03 '14 at 02:14
  • @devicenull too complex to discuss here but default maximum number of connections tracked (ip_conntrack_max) is 65k typically for servers with 1GB RAM or more, this just takes a relatively little amount of memory (about 22MB of RAM I think) and we can easily increment (double for ex) that 65k max conn tracking value. For an all-out DDoS hosing your server it becomes a data center issue and it doesn't matter much what you do in your iptables/server. – LinuxDevOps Apr 03 '14 at 16:07
1

Your question is a bit too general for an "answer", but I do have several suggestions:

First, use your hosts.deny file, either to allow only good guys or exclude selected bad guys. Remember to put your local networks in your hosts.allow file, which takes precedence. You can find bad guys in your Apache log, or by doing this (real example from today's log)

# BADGUYS=(`grep "authentication failure.*rhost=" /var/log/auth.log | \
    sed 's/.*rhost=\(\S*\).*/\1/' | \
    sort -u`)
# echo ${BADGUYS[0]}
109.228.29.199

and now block them

# for GUY in ${BADGUYS[*]}; do
> echo "ALL: $GUY" >> /etc/hosts.deny
> done

This might help enough to keep things working while your figure out the best firewall protocol. Disclaimer: The above snippet is a sample and not a full discussion or solution. As presented, it will probably block some legitimate users who just had trouble typing their password, so audit the results. This is why you want your local networks in hosts.allow.

Second, install DenyHosts or fail2ban, which do the above automatically. I have more experience with DenyHosts, but it seems to be falling out of favor. It targets hosts that attack your ssh service, but can block all services.

Third, solve the iptables rules issue. Ubuntu doesn't (or didn't) load rules by default, but there are several methods for making it persistent. Check out the iptables-persistent package.

nortally
  • 381
  • 2
  • 11