Getting a lot of hits on my web server and it's taking down my httpd. Possible DDoS?

Question

I'm getting a lot of hits on my server. This server normally gets little to no traffic yet there's is constant hits every time I bring the server back up. I get the following error first ip_conntrack: table full, dropping packet then sooner or later my httpd runs out of memory and my server becomes unresponsive. Any ideas on how to fix it?

latest head of my access_log. I changed http to hxxp

122.193.164.5 - - [27/Mar/2011:23:48:35 -0700] "GET hxxp://pubs.acs.org/templates/jsp/_style2/_achs/css/atypon-main.css HTTP/1.0" 200 174299 "hxxp://pubs.acs.org/doi/abs/10.1021/ac100095u" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; .NET CLR 1.1.4322; .NET CLR 2.0.50727; .NET CLR 3.0.04506.648; .NET CLR 3.5.21022)"

117.41.182.55 - - [27/Mar/2011:23:48:38 -0700] "GET hxxp://www5.tellgames.com/media/games/images/tellgames/120x90/02470dca7676598b9381e4c5dc2eef05.jpg HTTP/1.0" 200 4883 "http://us.tellgames.com/index.php?category=17&sortby=play&referer=ad2games" "Mozilla/4.0 (compatible; MSIE 4.01; Windows 95)"

117.41.186.191 - - [27/Mar/2011:23:48:37 -0700] "GET hxxp://s0.2mdn.net/1361550/K2147_NBRD_FYEA_728.jpg HTTP/1.0" 200 41371 "hxxp://ad.doubleclick.net/adi/N3340.161249.ADNETIK.COM/B5252096.3;sz=728x90;click=http://ad.z5x.net/clk?2,13%3B6b9391cec2a21533%3B12efb389ce8,0%3B%3B%3B2955295377,s5mFAKglGQBtfoAAAAAAAJJyIQAAAAAAAgAAAAYAAAAAAP8AAAABGB5.JwAAAAAAd0IfAAAAAABy8CsAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAABdhBAAAAAAAAIAAwAAAAAA6Jw4-y4BAAAAAAAAADY1YTAxMzY4LTU5MDctMTFlMC1iMTJmLTAwMzA0OGQ3NTRlMABwpioAAAA=,,http%3A%2F%2Fwww.providesearch.com%2F,;pc=[TPAS_ID];ord=[timestamp]" "Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.7) Gecko/20040614 Firefox/0.9"

173.252.208.155 - - [27/Mar/2011:23:48:38 -0700] "GET hxxp://ads.smowtion.com/st?ad_size=160x600&section=1739112 HTTP/1.0" 200 1336 "hxxp://www.consumerhealthdigest.info/category/health-information" "Mozilla/5.0 (X11; U; SunOS sun4u; en-US; rv:1.4) Gecko/20030701"

61.139.105.162 - - [27/Mar/2011:23:48:38 -0700] "GET hxxp://therugged.com/wp-content/uploads/2011/01/Steph61-80x53.jpg HTTP/1.0" 200 2980 "hxxp://www.therugged.com/category/lifestyle#player" "Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10.5; en-US; rv:1.9.0.1) Gecko/2008070206 Firefox/3.0.1"

The links in these logs are not on my server – user76112 Mar 28 '11 at 07:46 — user76112, Mar 28 '11 at 07:46

score 1 · Accepted Answer · answered Mar 28 '11 at 08:00

1

Are these domains you are hosting? I suspect not.

I've seen a big increase lately in scanning for open http proxies on my machines - it looks like you may running an open http proxy (which is just as bad as running an open mail relay - worse even, since most people now implement carious mitigations like RBL and SPF).

Disable proxying / add authentication / restrict to your LAN addresses.

OTOH if you really are webmaster for all these domains then have a look at mod_evasive and mod_security.

answered Mar 28 '11 at 08:00

symcbean

19,931
1
29
49

No these domains are not mine. – user76112 Mar 28 '11 at 08:16
I tried disabling all Proxy mods for apache. That didn't stop it. – user76112 Mar 28 '11 at 08:34
Thanks. I just followed this site http://wiki.apache.org/httpd/ProxyAbuse and I was able to deny the proxy with 403 Forbidden. – user76112 Mar 28 '11 at 09:15

score 0 · Answer 2 · answered Mar 28 '11 at 07:23

From the time-stamps, it does not seem like a very high hit-rate but from the IPs, it seems to be originating from all over. Most web-servers should be able to handle a few hits a second. However, you can try a few things to mitigate your problem.

If some of these connections are blocking the connections by holding onto an open connection, you could reduce the keep-alive time-out for each connection.
Check that your httpd is not consuming too much memory by reducing the maximum number of listening processes and threads.
Park your web-server behind a reverse proxy like varnish/pound and filter the destination connections at the edge, dropping invalid connections immediately.
Beef up your server to be able to handle the larger number of connections. Regularly test things by using siege or apache bench to ensure that you can handle a reasonable load.

1) I don't have keep alive on. 4) This web server should get a few hits a day. I'm having a hard time looking to reduce the number of processes and threads, any tips on what I should look up? — user76112, Mar 28 '11 at 07:44
#1 - you need to specifically turn it off, otherwise, it may be on a default value. #4 - it depends on your webserver, you will need to consult the documentation. — sybreon, Mar 28 '11 at 08:19

score 0 · Answer 3 · edited Mar 28 '11 at 11:21

Here's a little IPtables script that will limit the number of connections from individual IPs in a specified period. It's stopped this kind of thing for us in the past. It uses the recent module (which most modern OSs will load as an IPtables module automatically).

It does look as though those HTTP hits might not be causing the Web Server issue but this will just limit all traffic from an IP so won't be concerned with just HTTP hits if you remove the --dport option. As with anything IPtables related: use with caution (try it locally first if it's a remote Server).

iptables -F; iptables -X; iptables -nL

iptables -I INPUT -p tcp --dport 80 -m state --state NEW -m recent --set

iptables -I INPUT -p tcp --dport 80 -m state --state NEW -m recent --update --seconds 60 --hitcount 4 --rttl -j DROP

iptables -I INPUT -p tcp --dport 80 -m state --state NEW -m recent --update --seconds 60 --hitcount 4 --rttl -j LOG --log-level info --log-prefix IP-DROP:

Oh I forgot, this will help remove your conntrack errors and should be pretty harmless to increase "echo 131072 > /proc/sys/net/ipv4/ip_conntrack_max". YMMV. HTH, JR — Jonathan Ross, Mar 28 '11 at 09:02

score 0 · Answer 4 · answered Mar 29 '11 at 01:51

Use a reverse proxy, I would suggest Nginx because its light weight.

Setup a server section for your domain and one as a default for catch all requests. Any requests that arrive on the catch all domain return a 404 on them.

Also turn off KeepAlive by setting it to 0 and you probably want to disable logging for the default section till the attack goes down, else it will fill up our disk.

Getting a lot of hits on my web server and it's taking down my httpd. Possible DDoS?

4 Answers4

Linked