How to deal with this denial of service attack on an Apache server?

Question

I was the target of different attacks today. The last one created this traffic on port 80. Apache is down and the load on server remains high.

Firewall is enabled. Any suggestions?

15:27:10.203993 IP 188.125.110.42.38818 > 190.10.34.115.http: Flags [S], seq 3395115767, win 29200, options [mss 1460,sackOK,TS val 22777069 ecr 0,nop,wscale 7], length 0
15:27:10.204050 IP 103.21.182.66.62502 > 190.10.34.115.http: Flags [S], seq 3158122291, win 8192, options [mss 1460,nop,wscale 8,nop,nop,sackOK], length 0
15:27:10.204079 IP 50.143.120.99.54622 > 190.10.34.115.http: Flags [S], seq 3624243404, win 29200, options [mss 1460,sackOK,TS val 273756706 ecr 0,nop,wscale 7], length 0
15:27:10.204128 IP 62.216.187.156.42248 > 190.10.34.115.http: Flags [S], seq 846253572, win 29200, options [mss 1460,sackOK,TS val 523062741 ecr 0,nop,wscale 7], length 0
15:27:10.204305 IP 23.105.195.199.38172 > 190.10.34.115.http: Flags [S], seq 2631287484, win 14600, options [mss 1460,sackOK,TS val 2125155826 ecr 0,nop,wscale 7], length 0
15:27:10.204774 IP 76.111.172.248.39602 > 190.10.34.115.http: Flags [S], seq 2199780458, win 29200, options [mss 1460,sackOK,TS val 249941894 ecr 0,nop,wscale 7], length 0
15:27:10.204797 IP 104.154.65.106.44962 > 190.10.34.115.http: Flags [S], seq 2397138535, win 28400, options [mss 1420,sackOK,TS val 1992688634 ecr 0,nop,wscale 7], length 0

Edit: This was our solution to the problem.

Answer 1

This is a classic SYN flood attack, preventive measures include

• hashing syn packets, implementing syn cookies
• checking if the client properly sends RST after an invalid packet sent back to him
• ddos mitigation software often offer to time out specific syn packets based on advanced filtering
• not storing the whole syn packet for further reference, only the IP and sequence number (also requires additional firewall/software to be installed on your part)

Answer 2

Limit NEW traffic

sudo iptables -A INPUT -p tcp --dport 80 -m state --state NEW -m limit --limit 50/minute --limit-burst 200 -j ACCEPT

Limit established traffic

sudo iptables -A INPUT -m state --state RELATED,ESTABLISHED -m limit --limit 50/second --limit-burst 50 -j ACCEPT

Enable syncookies on your kernel ( Recommanded by @Rápli András)

echo 1 > /proc/sys/net/ipv4/tcp_syncookies

Install and configure ddos-deflate

Notable Features:

• It is possible to whitelist IP addresses, via /etc/ddos/ignore.ip.list.
• It is possible to whitelist hostnames, via /etc/ddos/ignore.host.list.
• Simple configuration file: /etc/ddos/ddos.conf
• IP addresses are automatically unblocked after a preconfigured time limit (default: 600 seconds)
• The script can run as a cron job at chosen frequency via the configuration file (default: 1 minute)
• The script can run as a daemon at chosen frequency via the configuration file (default: 5 seconds)
• You can receive email alerts when IP addresses are blocked.
• Control blocking by connection state (see man netstat).
• Auto-detection of firewall.
• Support for APF, CSF and iptables.
• Logs events to /var/log/ddos.log
• Uses tcpkill to reduce the amount of processes opened by attackers.

Answer 3

SOLVED

In this case, it was a synflood attack with compromized Wordpress sites that generated tons of traffic (pingback) to our site at Gbytes/hour.

Here is the full description of the attack. In the referrer/header you can read "Wordpress" as the user agent: https://www.hyperborea.org/journal/2014/03/pingback-ddos/

Blocking these IPs with a firewall rule (see list in the link) solved the problem. I share the list, without any responsibility of curse.

LIST OF 5000 IPs involved to be blocked with firewall.

Answer 4

The output you are showing suggests you are receiving a flood of SYN packets and not responding to any of them. Extrapolating from the numbers shown it amounts to about 7.5kpps. Even assuming a maximum packet size of 60 bytes of IPv4 header and 60 bytes of TCP header that still totals less than 8Mbit/s.

So if your network connection is 10Mbit/s or faster you should be able to sink that traffic assuming you have sufficient processing power to handle the number of packets per second.

From the limited information in your question it is impossible to tell what is causing the current load. It is a possibility that your firewall rules are structured in a way that cause each packet to consume a significant amount of CPU time in the firewall. If that's the case you need to restructure your firewall rules to be more efficient.

It seems you have configured a firewall to silently drop all of the packets. That can actually make the problem worse because legitimate clients will retransmit their SYN packets until you respond (or the client times out).

Rather than silently dropping the packets I recommend that you respond to as many of them as your upstream bandwidth permits. The responses would need to be either SYN-ACK packets or RST packets.

If you are experiencing a SYN flood where you cannot tell the difference between attack packets and legitimate packets, and if you need to keep serving requests from legitimate clients, I recommend that you enable SYN cookies as suggested in the answer by Rápli.

Under such circumstances SYN cookies can be the difference between your server responding slowly under the load or not responding at all.

Don't block IP addresses based on just the SYN packets. If you only see a SYN packet the source IP may be spoofed. And blocking based on that will do you no good and open you up to a different kind of DoS attack.

If a client completes a handshake and then sends bad traffic you can consider blocking the IP address. However doing so with IPv4 can lead to collateral damage, so it should be done only as a last measure. If you do block traffic that way make sure that you respond with RST packets, and make sure that your firewall rules are structured in a way that allow them to be evaluated with minimal CPU consumption.