6

Last week my network was hit by a DDoS attack which completely saturated our 100 MBps link to the internet and pretty much shut down all the sites and services we host.

I understand (from this experience as well as other answers) that I cannot handle a DDoS attack such as this on my end, because even if we drop the packets they have still been sent over our link and are saturating our connection.

However when this happened my ISP was (strangely enough) unable to tell me where the attack was coming from. They said if I could determine the source (E.G. via tcpdump) I could give them IP addresses to block. But things were so overloaded that running tcpdump was impossible. I just couldn't view the output.

Nearly all our servers are behind a pfSense router. How can I detect a DDoS attack using pfSense so I can tell my ISP who to block? I don't want to block the attack myself, I just want to get alerts / be able to view a list of IP addresses that are using way more bandwidth than normal.

The pfSense router is running Snort, if that can be used to assist in any way.

Community
  • 1
Josh
  • 9,001
  • 27
  • 78
  • 124
  • 4
    If it truly is a DDOS, you're going to end up with a LOT of IP addresses. That being said, pfSense does have 'Packet Capture' option under Diagnostics. – charlesbridge Aug 03 '12 at 18:08
  • Rather then investigating the source, I would investigate the target. The first D in DDOS stands for `Distributed`. This means it could be coming from virtually everywhere at once. If you are able to get the target and let the target be null-routed by your ISP, you'll be better off. – Bart De Vos Aug 03 '12 at 18:09
  • @BartDeVos but that would mean that server would be taken offline, no? – Josh Aug 03 '12 at 18:11
  • 1
    @Josh: Yes. If it's a DOS-attack you just get the IP and block it (at ISP-level if you want). When you have a DDOS, this is nearly impossible. How will you be able to distinguish 'DDOS-traffic' from 'real traffic' ? – Bart De Vos Aug 03 '12 at 18:13
  • @BartDeVos I would assume it would be the top IPs by bitrate... Maybe I should be asking how to defend against a DDos. Besides isn't a DoS attack from more than one Ip technically a DDoS attack? :-) – Josh Aug 03 '12 at 18:40
  • Your assumption is not valid. It is a DISTRIUTED attack. Lets ask for 1 page per minute. With 600.000 machines. Have fun. – TomTom Aug 04 '12 at 09:02
  • @TomTom and that's the kind of attack which brought down Estonia. The attack which brought down my network last week was ~5 machines. The difference in my network graphs was *clearly* noticeable, as my incoming traffic increased tenfold the normal rate. – Josh Aug 04 '12 at 12:55
  • Actually no ;) 5 machines is NOT a DDOS. A DDOS is done by a bot network that anyone can rent for quite cheap these days and that HAS many machines. 5 machines is a DOS - a normal Denial of Service, nothing really distributed about it. – TomTom Aug 04 '12 at 16:02

1 Answers1

14

There are several different types of DDoS so any generic information about them may only be correct for one particular type. For instance, the idea that a DDoS always exhausts your bandwidth is incorrect. What you need to do is analyse (some of) the traffic, determine why it's breaking your site, find a way of identifying it and then decide on an action that can block the traffic.

DDoS traffic probably doesn't look like real traffic but what makes it different is not necessarily the quantity. During a recent DDoS we had, our hosting provider decided to block the top ten highest connecting IP addresses. These, of course, all had nothing to do with the DDoS and one of them was actually the Google bot. They would have blocked our office except that they recognised the IP address. The standard advice to use netstat, sort and uniq to find the highest connections is not necessarily good advice.

Our DDoS was a SYN-flood, which means a couple of things for detection:

  1. These are not fully opened connections so depending on how you're measuring connections you might not even see these.
  2. There is not much identifying information to go on. Pretty much just source IP address and source port.
  3. The source address can (and almost certainly will) be spoofed because they're not expecting a response.
  4. You may get lucky and find an unusual TCP flag set.

You might only see each IP address once which would make blocking an IP address you had already seen fairly pointless. In practice, we saw 100,000 unique source IP address over 10 minutes and 140,000 packets per second which means (on average) each IP address came back about once per second. Blocking IP addresses would have been effective in our case. The total bandwidth was only 70Mb/s and since the source addresses were spoofed, the whole DDoS could easily have been pushed out by a single server somewhere.

Since each IP address was only sending roughly one packet per second, this was considerably lower than pretty much any other legitimate IP address. Many of our website home pages are up to a MB in size, meaning thousands of packets for what the user sees as a single request.

Mostly the source ports were randomised but for a few hours it changed and the source port was always 1234. This made it very easy to identify the traffic.

In our case, the problem was that the number of packets per second was greater than what the firewall could handle. Blocking these IP addresses at our firewall wouldn't have worked because the firewall was the problem but if the traffic is getting through the firewall and the web boxes are the problem, blocking at the firewall can help. Our hosting provider was able to block the traffic upstream from our firewall when the source port switched to 1234.

There are other types of DDoS and some of these fill your bandwidth. Ours knocked out our firewall anyway, meaning we couldn't access any part of our infrastructure. For your specific question, you will need some kind of out-of-band way to communicate with your firewall and/or boxes so that if your internet pipe gets filled, you can still get in to diagnose the problem. This is generally a good idea anyway, because there are so many other ways your pipe can go down. I tend to see a lot of ADSL modems in other people's racks when I walk around data centres and I wouldn't be surprised if some of these were for out-of-band communication.

Two other DDoS types I have seen are reflected DNS requests and expensive HTTP requests. The reflected DNS requests are particularly troublesome if you run a DNS resolver because you don't want to block these IP adresses in case you want to receive a real answer from one of them. I would change my hosts to use a new DNS resolver (maybe a free public one) and block all other DNS traffic.

The expensive HTTP requests tend to be targeting your CPU/Memory/IOPS. They also come with a lot of identifying information such as HTTP headers and the source addresses can't be spoofed. A tool like mod_security can do some amazing things identifying and blocking these types of requests. It can even drop the tcp connection rather than sending back an HTTP response.

In short:

  1. Get out-of-band access.
  2. Know your protocols inside out. (TCP, IP, HTTP, whatever you use.)
  3. Know your tools inside out. (pfSense, tcpdump, mod_security, etc.)
  4. Know your options. (HTTP 403, pfSense block, ISP firewall block, null route, etc.)
Ladadadada
  • 25,847
  • 7
  • 57
  • 90