1

I am running a site with 2 GB ram on centos. Using LAMP

Everyday for a few hours (about 3 to 4 hours) apache goes down. It takes too long to load a page and most of time it shows Request time out. But other ports are okay like ftp also 7778, as I am using kloxo.

I am not a server expert but I have googled and done a bit research and I think this is a ddos syn attack.

I used netstat -n | grep :80 | grep SYN |wc -l to look into syn connections

During the attack it shows a number more than 3000. But when my site backs to normal it shows a number around 100.

So please help me. and tell me how to stop this this is really pissing me off.

Suman Biswas
  • 49
  • 2
  • 2
  • 4
  • Try one of the solutions from [this question](http://serverfault.com/questions/230692/how-can-i-prevent-apache-dos-flood): mod_evasive or fail2ban. – Russ Amos Aug 09 '12 at 20:02
  • <3 mod_evasive! – Dave Drager Aug 09 '12 at 20:05
  • 2
    What state are the 3000 connections in? Your grep doesn't distinguish between `SYN_SENT` and `SYN_RECV`. The first means you are making a lot of connections on port 80 to other people, the second means other people are making a lot of connections on port 80 to you. (Also, technically either of these could be *from* port 80 but that would be a bit weird.) If you take off the `| wc -l` from the end you will get this information and also the list of IP addresses. – Ladadadada Aug 09 '12 at 20:54
  • That's not a SYN DDoS, it's just your site being overly popular. A real SYN flood would knock out all TCP ports on the machine. Tune your Apache config and system resources to be able to handle the traffic you're receiving. – womble Aug 09 '12 at 23:38
  • Is it helpful to suggest that the most effective way to prevent is any DDoS attack is not to piss off Anonymous or other script kiddies? (And/or not be on the same network as anyone who does...) – HopelessN00b Aug 10 '12 at 00:31
  • @womble But my site get more visits when it is not under attack and works fine with the current configuration. During the attack number of connections goes down `netstat -an |grep :80 |wc -l` . Google analytics also shows a low number of active visitors during attack. Only syn connections goes high, as I mentioned above. – Suman Biswas Aug 10 '12 at 06:41
  • Your symptoms still don't match a SYN flood attack. The elevated number of `SYN_RECV` connections (incidentally, you don't even *know* they're `SYN_RECV` because you're just grepping for SYN, which includes `SYN_SENT`) is just because you're out of workers to actually process the requests. – womble Aug 10 '12 at 07:18
  • @womble I have checked today. during the attack SYN_RECV connection was more than 1000 and SYN_SENT was only 36 . Now I think this is a SYN flood attack. Please help me. I also tried to check those ips. But there was no common pattern, except they were like this 42.106.175.xxx . – Suman Biswas Aug 10 '12 at 15:23
  • It. Is. Not. A. SYN. Flood. Attack. 1000 SYNs is *nothing*, and wouldn't stop your system from working. Your problem is that you aren't accepting connections fast enough, because you aren't processing requests fast enough. That is *all*. – womble Aug 11 '12 at 11:04
  • I would say these symptoms do indicate an attack, but when you see them, you're not the victim. You're just a reflector, maybe seeings some collateral damage. The source address of the SYN packets is the faked address of the victim, and your open port 80 is sending up to 5 SYN/ACKs there. What that does leave behind is a lot of half-open connections. Try `echo 1 > /proc/sys/net/ipv4/tcp_syncookies` To avoid amplifying the attack, lower net.ipv4.tcp_synack_retries to 2, or maybe use iptables -m connlimit --connlimit-upto 50 – Cedric Knight May 10 '15 at 21:30

4 Answers4

8

The first thing to do is to determine what's happening. All you know at the moment is that Apache is unresponsive sometimes and during these times you have a lot of half-open TCP connections. You don't know if the TCP connections are the cause of the problem or just a symptom.

Do you have any performance monitoring system such as Cacti, Munin, Zabbix, Observium or any of the other options in this space? If not, get one of them now. Configure them to graph all the normal things such as memory usage, load average, CPU usage, free disk space, IOPS, network usage, etc but it also might be worth adding custom metrics such as requests per minute or TCP states. Also add whatever templates match the services you are using, such as Apache, MySQL, memcached, etc.

By analysing the graphs that these tools produce you should be able to find the resource that is hitting 100% during these downtimes. From there, you can trace the cause-and-effect chain back through to the initial trigger that caused the problem. It's possible that the resource that is being exhausted is not being measured by you and may not even be in your control.

As a guess, if your connections are in the SYN_SENT state, you are probably using someone else's API over HTTP and either they are down or they temporarily blocked you. If they are in the SYN_RECV state, there might be a firewall problem that is blocking the SYN-ACK responses from being received by the client or the response from the client being received by you. It could also be the SYN-flood you suggested.

During a real SYN-flood, you will see the bandwidth and the number of packets per second jump significantly. Use tcpdump -w packet-capture-file.cap or -j LOG in iptables to log these packets and see if you can spot a pattern. Maybe the source address is always the same, maybe it's always in a small range, maybe a strange TCP flag (such as URG or PSH) is set. Failing that, see if the IP addresses involved make multiple connection attempts. If they do, you can add them to a DROP rules in your firewall so that your OS won't have to deal with them. Depending on how many different IP addresses there are, you may need to split your rules into several tables to reduce the size of the list iptables has to scan through.

The DDoS may be big enough that it's actually exhausting your hosting provider's bandwidth. If that is the case, they or you will probably need to go to a dedicated DDoS mitigation company. These guys use either DNS or BGP to route your traffic to them, filter out the DDoS traffic and send the rest on to you. They're generally not cheap and you will have to weigh the cost of downtime against the cost of the mitigation service. There are also services like CloudFlare that can prevent DDoS attacks up to certain limits and have more affordable pricing plans.

Ladadadada
  • 25,847
  • 7
  • 57
  • 90
  • +1 for explaining that the OP has no idea exactly what the problem is without ay monitoring analysis. – Jim B Aug 09 '12 at 21:52
  • Thank you for your reply. It never uses a high resource, either it is under attack or not. cpu and ram uses always under 40%. For a first few days I was really confused I found nothing odd in the server, but apache was responding very slow for those few hours. And after that I have noticed during that time SYN_RECV connections goes very high. Do u think this is a ddos attack ? more suggestions needed. – Suman Biswas Aug 10 '12 at 06:57
  • There's rarely any doubt when you get a real DDoS. The bandwidth spike is massive and you usually can't access your server at all unless you have an out-of-band access method. Keep investigating and check those monitoring graphs. – Ladadadada Aug 10 '12 at 20:18
1

Are they distributed or coming from a single IP? Either way, I would recommend a firewall such as csf for restricting TCP syn attacks.

Dave Drager
  • 8,315
  • 28
  • 45
1

A quick and cheap solution is to:

  • Reduce the timeout on the syn packets timeout

    Edit /etc/sysctl.conf

net.netfilter.nf_conntrack_tcp_timeout_syn_recv=30

  • Limit the amount of syn packets you can receive

create new chains

iptables -N syn-flood

limits incoming packets

iptables -A syn-flood -m limit --limit 10/second --limit-burst 50 -j RETURN

log attacks

iptables -A syn-flood -j LOG --log-prefix "SYN flood: "

silently drop the rest

iptables -A syn-flood -j DROP

Source

ponsfonze
  • 384
  • 2
  • 11
  • 1
    Depending on how much traffic your site gets or if your site is in constant communication with various servers (RSS feeds, etc), this rule could actually stop legitimate traffic. Also, just because you limit how many SYN pps you receive will not eliminate a DoS situation. Even if you --limit 10/second your system is still waiting for the ACK from the requester. So really, you are just delaying the DoS not mitigating or eliminating it. For a holistic approach, Id look into blacklisting based on the logged IP's. I still like the rules though Ponsfonze! – Blake Aug 09 '12 at 22:52
0

Who cares about tracing the IP's. 90% of DoS/DDoS attacks will be proxied! You need to setup a blacklist period based on the "malicious traffic".

http://perishablepress.com/eight-ways-to-blacklist-with-apaches-mod_rewrite/

In a SYN attack your system is flooded with a series of SYN packets (obviously). Each packet causes your system to issue a SYN-ACK response. Your system waits for an ACK that follows the SYN+ACK (3 way handshake). Since the attack never sends back an ACK, your system resources get consumed. Once the queue is full your system will ignor incoming requests from legitimate users for services (http/mail etc). Hence it is necessary to stop this attack with iptables, ipfw, etc.

As far as a iptables rule:

iptables -A INPUT -p tcp ! --syn -m state --state NEW -j DROP

Blake
  • 122
  • 1
  • 5
  • 1
    Turning on syn-cookies is the appropriate response to the OS not having enough memory to keep track of the half-open connections. Blocking with mod_rewrite is ineffective for SYN-floods as Apache doesn't see a request until after the TCP connection has been completely opened. – Ladadadada Aug 10 '12 at 20:13