3

Site are crawled by anonymous bot hosted on amazon ec2. This robot doesn't respect robots.txt and creates high load on web server so I added check if reverse IP for request ends with "amazonaws.com" then server returns 403 page immediately.

This solved problem but may be it can cause other problems? ec2 may be used for some "good" bots and this will cause access problem for theirs. Can you give example of such problems?

valodzka
  • 177
  • 3
  • 10

2 Answers2

5

Amazon EC2 is a hosting platform. They don't directly control what people host. If you block the whole *.amazonaws.com domain then you will stop access to any hosted service using EC2. Which is quite a lot these days.

George Hewitt
  • 1,066
  • 7
  • 13
1

Check out this similar question: it shows how to block by user agent directly in the .htaccess file. This is good for robots that fail to follow your robots.txt rule...

Blocking by user-agent string in httpd.conf not effective

And you would put that in either the httpd.conf file, OR a .htaccess.

Good luck.

Community
  • 1
U4iK_HaZe
  • 631
  • 5
  • 13
  • 2
    If a robot is evil enough to ignore a robot.txt, why do you believe that it wouldn't just provide a forged agent so it appears to be Internet Explorer? – Zoredache Sep 15 '11 at 19:35
  • Because with .htaccess you block by user agent and/or IP, and doing such, do you think a robot will make more than 5 user agents? If it's actually a robot, it doesn't care who you are. It just goes. It's not going to make a new rule just for your site because it's blocked. AND plus: if it does choose the IE bot, he can just contact Amazon and let them take care of it. I'm sure he's not the ONLY one with this issue. – U4iK_HaZe Sep 15 '11 at 19:41