Apache/Linux server, DoS attack from own IP

Question

I have an unusual problem I've been trying to diagnose for a while:

• It's about a Debian server running a custom compile of apache 2.2 with PHP, Red5, MySQL 5.5 (standard binary), sendmail (distro version), and crashplan.

• Every other day I see a high amount of HTTP requests to random files, mostly images - we're talking upwards a thousand concurrent connections.

• These requests come from the servers own IP address (!).

• It's usually a limited set of files that's requested over and over again. I see no real pattern, but it doesn't look like someone scraping information, it looks like a DoS attempt.

• Cron runs a script that temporarily bans IPs with more than 200 connections, so this is usually curbed before it can get really problematic. After 1-3 10 minute bans, the attack usually stops.

• This has been going on for months. Since the attacks are caught and curbed, I completely fail to see the point.

• It's happening at random times and intervals, but usually around morning times UTC.

• No referer or agent is being sent with these requests.

I've checked the webserver and red5 logs for related requests around the same time, in case a script on the server is abused to send queries to itself, but couldn't find anything. There's nothing in the apache errors logs or syslog around that time. Rkhunter didn't find anything out of the ordinary either. The server doesn't source route packets, so spoofing shouldn't be an option either.

I'm at a complete loss as to the method and the reason. Any ideas what to check would be greatly appreciated. :)

UPDATE: Following Isernis advice, I've prepared a mechanism to catch some information on the next occurrence. This is (a slightly generalized version of) the method: http://pastebin.com/6uSUKVbh

ANSWER: This is a social media site allowing mySpace type profiles utilizing FCK Editor. Since that is a bit of a security nightmare, the profiles posted by users undergo extensive checks, one of which probes links/images posted. For one I did not exclude the sites own domain in these checks and two due to a bug related to redirects, every link or image was hit 10 times instead of once. So when a user with a profile containing extensive linkage to the site itself hit the save button, the site would DoS itself. :P In particular this concerned one user who has a bazillion items in her profile and tends to save often.

Thanks to Iserni for the right idea how to diagnose this issue!

ANSWER EDIT: I was wrong about the bug. She actually does have some images 10 times or more inside the profile. More specifically near 1000 links and images to be checked each save. I didn't see that coming. :P

Answer 1

First thing: find out where do these requests come from. It has to be a local process, nothing else is likely to be able to spoof a TCP handshake on a modern Linux platform (nothing, that is, that would then proceed to waste such a feat on requesting random images).

If there are recurrent URLs, you can shadow them behind a RewriteRule so that any such request will actually trigger a script. In the script you can run additional checks to see whether the request is legit (and you will then output the proper headers just as if it was the image the legit client expects), or if it is one of the bogus requests. Against the bogus request you can log e.g. the incoming port. Armed with that, you can query netstat and find out the process. You can also run ps and inspect all active processes in the instant of the bogus request.

I am quite sure that the culprit will prove to be Apache itself (I once had a "cache priming" script go rogue on me due to a vhost modification - I had forgotten putting the script in crontab - and got really weird symptoms, somewhat like yours, until it all came back to me; but your case feels different).

To further refine the scene while containing costs, you can add PID/TID to Apache's CustomLog. Then you will be able to cross-check the requests received from the Apache child gone rogue.

Another possibility is to determine exactly how these requests are made. If through Apache, this means fopen_wrappers, cURL, socket functions, or maybe shell utilities (these should both appear in ps output and result in a much more massive server overload, though). You can prepare a series of script that will:

• gracefully restart Apache without any changes
• " " , disabling temporarily one of those functions
• " " , re-enabling same

After verifying (just to be sure) that a restart does not fix the problem (if it did, it would be a quite different can of worms), you can proceed to temporarily disable - a couple dozen seconds each, no more - one function after another. Suppose that disabling curl results in the system immediate return to normal: then you could restrict investigations to scripts using cURL, and maybe even wrap the cURL function with a logging wrapper.

In case the guilty party turns out not to be Apache, still you will be able to determine what is doing this; then either reinstall the affected program (even if I find it unlikely for any random anomaly to turn a program into a repeat-HTTP-GET-requestor) or inspect its configuration, ancillary data files, scripts, and so on and so forth, looking for any difference from a known clean installation. Since I don't usually believe in gremlins, I expect for some difference to stand out in the end.

Answer 2

Unix (and Linux) has a wealth of tools for analysing stuff like this. My first stop would be to grab the output of netstat -nap e.g. on my local machine...

Proto Recv-Q Send-Q Local Address               Foreign Address             State       PID/Program name
...
tcp        0      0 192.168.0.2:80              192.168.0.2:59875           ESTABLISHED 5281/httpd
...
tcp        0      0 192.168.0.2:59875           192.168.0.2:80              ESTABLISHED 32588/chrome
...

Here I can see than chrome (pid 32588) is connected to port 80 / httpd (pid 5281). Since this is a pre-fork installation of apache, I can get more information about the httpd process by logging %P or by looking in /proc/5281/fd (latter will probably require scripting to grab the data at the time of the request).

This will allow you identify the client process.

The most likely candidates are a badly configured proxy or buggy code.

Answer 3

If this were my server, I'd be running an strace on Apache. Running this on every child process in prefork mode can be quite disk intensive, especially when your server is already being overloaded. You do have to keep an eye on your disk space as well, because if it runs out, Apache stops serving requests.

Make sure you use a snaplength long enough to capture the entire request: -s 400 should do.

If Apache is making requests to itself, any GET string will appear in the strace dumps for two different PIDs: one that made the request and one that received it. In the one that made the request, you want to find the request that it received and was processing when it made the request to itself.

I normally do something like this:

for x in `ps -ef | grep apache | awk '{print $2}'`; do strace -s 2000 -p $x -o trace.$x & done

If you want to limit yourself to a subset of Apache children for performance reasons, add a head in there:

for x in `ps -ef | grep apache | head | awk '{print $2}'`; do strace -s 2000 -p $x -o trace.$x & done

But be aware that this makes it less likely for you to capture what's happening.

Make sure you have two SSH sessions open as all those backgrounded tasks can still write to your session. When you want to stop stracing, either restart Apache or run this in the other one:

 for x in `ps -ef | grep strace | awk '{print $2}'`; do kill $x; done

---

My gut feeling on this one is a "static" module written in PHP that pre-processes images (resizing them for instance) before sending them to the client and it does this with include($image). If $image happens to contain an image URL from your own site rather than a file path from the local filesystem, recursive requests are the result.

It could be using the curl() functions rather than include().

Answer 4

It sounds like a typical DOS attack. They are probably hoping to get the server to respond to a request from itself and hoping to get a loop like the "ping of death". It's also a convenient way to spoof to get around some firewall rules and cause general headaches. Blocking the external IP at the firewall is probably the best bet so that they can't get the requests in the door.

Answer 5

It sounds like a DDoS attack which is spoofing the IP of the server. The best action would be to put a packet filter on the external router rather than using firewall rules, as using the router will reduce load on the firewall. On a cisco router the simple solution would be to write an access list with the source being your public block, and destination being any, then applying it to the external interface as an "ip access group in".

Rate limiting ICMP would be a good idea too, they may try to ping flood you.

You should give some thought to rate limiting valid traffic as well, the next DDoS attack won't be spoofing your own IPs, they'll be using valid IP addresses that you cannot filter without filtering out your customers. A well chosen rate limit will keep your server from running out of resources.