1

Okay, an Ubuntu server I manage fell victim to a DDoS attack today. Usually this is unpleasant but not that big a deal. A few high server load moments & then it passes. Today was clearly different. For the record, I have years of server attack scars. But today felt different. Read on.

The solution was solved by re-enabling a ModSecurity rule that prevents curl access via a call to a pure IP address. Meaning if a hostname is mygreatsite.com and the IP address is 123.456.78.90 and I have a virtual host for mygreatsite.com & that is enabled. And ModSecurity is active with the rule that prevents curl access via a call to a pure IP address, then a visitor would get the proper content when they visit mygreatsite.com, but are blocked by ModSecurity with a 403: Forbidden if they attempt to access the site via 123.456.78.90. Great problem solved!

But while the attack was happening, three things were clear:

  1. Apache Processes Through the Roof: The Apache process load as reflected in Munin was through the roof & consistently 3x higher than normal.
  2. No Corresponding Web Traffic Logged: There was no corresponding traffic logged via Apache & reflected in AWStats. This was the case for a few hours. This is across all logs on all virtual hosts on the server including the default virtual host that would be connected to a bare IP address.
  3. Server Load Was Moderately High, But Not Typical Attack Level: While overall server load was high, it was only in the single digits—4, 5, 6, 7, etc…—and not a typical DDoS where loads can go double and even triple digit like 20, 30 & even 120 (!!!).

To me item number 3 was the most unusual part of this attack. I have not experienced a DDoS where server load was basically equivalent of a heavy traffic day & not an insane server load connected to the attack.

Additionally, running elinks to fetch Apache server-status from the command line showed the connection pool filling up like crazy within minutes of a restart:

elinks http://localhost/server-status?refresh=1

And past any of that, I just noticed an Ubuntu security patch today specifically for curl issues. And this specific issue piqued my interest:

libcurl incorrectly validates wildcard SSL certificates containing literal IP addresses.

So here is the question: What the heck happened? I know re-enabling a ModSecurity rule stopped the attack, but could it have in any way been connected to the libcurl issue? And if so, how can I detect or prove that? And past that if the issue was with libcurl—which I assume is coming from weaponized clients using the flaw—then does that mean that someone can simple keep a hacked older version of curl around to DDoS again? And writing that, any hacker can just download source code & patch it to hack so why the flood today?

My guess is the patch to curl made more people aware of the flaw & the “enthusiasm” of script kiddies around the globe brought a mad flood of scanning around within hours of the exploit being found? But again, this confuses me because it’s on the client curl install, correct? I can’t imagine curl on the server side would be victim to this.

Giacomo1968
  • 3,522
  • 25
  • 38
  • 1
    possible duplicate of [I am under DDoS. What can I do?](http://serverfault.com/questions/531941/i-am-under-ddos-what-can-i-do) – Jacob Apr 16 '14 at 03:13
  • 2
    It's not really a DDoS if you can stop it on your end. – Jacob Apr 16 '14 at 03:13
  • Are you sure that the attack traffic isn't showing in the Apache logs? The "main" vhost might have a different access log than the default vhost which is getting these requests? – Shane Madden Apr 16 '14 at 03:15
  • I appreciate the duplicate flag, but not to invoke Passover too much, but to me & my experience this attack was different from every other attack I have experienced. The timing of the `curl` patch & my re-enabling of the related `ModSecurity` rule connected to `curl` seems oddly too perfect. – Giacomo1968 Apr 16 '14 at 03:15
  • @ShaneMadden I assure you that all server logs were checked & I have AWStats grinding through all logs on the server. 100% no correlating traffic. – Giacomo1968 Apr 16 '14 at 03:15
  • @Jacob “It's not really a DDoS if you can stop it on your end.” Nonsense. You can stop DDoS since there are differing levels of skill in folks who engage in DDoS. – Giacomo1968 Apr 16 '14 at 03:18
  • 1
    It looks to me like it was a SYN attack. It is only a fraction of a connect to your webserver, meaning that it will deplete the amount of available connections. It being only a SYN request and perhaps some more data, as opposed to a complete HTTP request, it would not show up in the apache logs. – Jarmund Apr 16 '14 at 07:10
  • @Jarmund If it were a SYN attack, then why would a simple activation of a ModSecurity rule—which operates as a module in Apache—block it? Wouldn’t a SYN attack be at a lower level of network traffic? – Giacomo1968 Apr 16 '14 at 09:21
  • 1
    Just because somebody _can_ launch an attack, it doesn't mean they have enough resources to take you down. – Michael Hampton Aug 14 '15 at 01:50

0 Answers0