block fake google bots

Question

How could I block DDOS attacks with fake Google bots?

I found 2 solutions on the net. But both seems to block also correct google bots.

# Block fake google when it's not coming from their IP range's (A fake googlebot) [F] => Failure
RewriteCond %{HTTP:X-FORWARDED-FOR} !^66\.249\.(6[4-9]|[78][0-9]|9[0-5])\.
RewriteCond %{HTTP_USER_AGENT} ^Mozilla/5\.0\ \(compatible;\Googlebot/2\.[01];\ \+http://www\.google\.com/bot\.html\)$ [NC]
RewriteRule .* - [F,L]
# End if match

And here is the second one:

# Validate Googlebots
RewriteCond %{HTTP_USER_AGENT} ^Mozilla/5\.0\ \(compatible;\ Googlebot/2\.[01];\ \+http://www\.google\.com/bot\.html\)$
RewriteCond %{HTTP:Accept} ^\*/\*$
RewriteCond %{HTTP:Accept-Encoding} ="gzip,deflate"
RewriteCond %{HTTP:Accept-Language} =""
RewriteCond %{HTTP:Accept-Charset} =""
RewriteCond %{HTTP:From} ="googlebot(at)googlebot.com"
RewriteCond %{REMOTE_ADDR} ^66\.249\.(6[4-9]|7[0-9]|8[0-46-9]|9[0-5])\. [OR]
RewriteCond %{REMOTE_ADDR} ^216\.239\.(3[2-9]|[45][0-9]|6[0-3])\.0
# Optional reverse-DNS-lookup replacement for IP-address check lines above
# RewriteCond %{REMOTE_HOST} ^crawl(-([1-9][0-9]?|1[0-9]{2}|2[0-4][0-9]|25[0-5])){4}\.googlebot\.com$
RewriteRule ^ - [S=1]
## Block invalid Googlebots
RewriteCond %{HTTP_USER_AGENT} Googlebot [NC]
RewriteRule ^ - [F]
# END sending now 403 to fake Googebots

Could anybody suggest a solution to block DDOS attacks with fake googlebots?

Don't do this in .htccess it's going to be slower than a slow thing towing a lead balloon. When under DDOS don't add to it yourself. — user9517, Nov 02 '15 at 20:09

QuentinMoss · Answer 1 · 2015-11-03T01:41:04.130

The two methods that you provided appear to allow based on a known range of IPs. You'll want to verify that the regex match is up to date and that the range you're checking is what Google is actually using. If you're actually under a "DDoS", I do not suggest using the reverse DNS lookup option in your .htaccess file.

Alternatively and suggested, you could use some type of "web application firewall". Sucuri and CloudFlare both offer a service to filter this type of traffic (and other malicious stuff) and can greatly reduce the number of 'bad bot' requests that you serve.

-- CloudFlare: https://www.cloudflare.com/waf/

-- Sucuri: https://sucuri.net/website-firewall/

score -2 · Answer 2 · answered Nov 02 '15 at 19:24

-2

This is where a firewall comes in. Firewalls have special built-in purposes, some of which are for this very thing.

I would look into ASA or Dell SonicWall firewalls. By default, it would block something like that out anyway.

answered Nov 02 '15 at 19:24

Jonas Lear

460
1
3
5

1

You're not the only person to think this... But no, once your firewall has been reached, your bandwidth has been used up. And the DDOS was effective. – Reaces Nov 02 '15 at 19:32
@Reaces Depends on whether the DDoS is a pipe-filler or a CPU-filler. It's often easier to nuke a site by just making "expensive" calls to the webapp than to spam it with traffic. – womble Nov 04 '15 at 05:43

block fake google bots

2 Answers2