2

We are noticing that a significant amount of web traffic is from content scrapers (determined due to their crawling pattern). They are useless visitors to us but consume a lot of our resources (bandwidth, cpu). Is there any application/firewall to detect content scrapers and block them?

Excluding Search engine crawlers, they are not useless.

Note: I prefer to use existing solutions. It believe this is a common problem and there should be an existing solution.

newbie
  • 93
  • 4

1 Answers1

0

The best way to do this is to block the traffic using netfilter/iptables as this is essentially more performant than blocking via apache2 / php. Problem here is that you're required to know the ip / hostname of the content scrapers.

A possible extension could be you try to detect content scrapers based on their behaiviour (-> statistical methods! - e.g. requests per minute) or e.g. search for missing useragent or other stuff a normal user browser would have and then deny access for them. Of course you could also add the IP / Hostname via php (or whatever environment you use) to iptables so it's blocked. But normally this requires root permission and it's NOT a good idea to give root permission to your apache2.

Simon Strasser
  • 227
  • 1
  • 8
  • Content scrappers normally provide a legitimate user-agent. Request per/min can not be a good measure to block the IP as some offices have several users behind a proxy/ISA. – newbie Dec 31 '12 at 15:00
  • I use a system as described above for a customer of mine - and it works fine since half a year :) – Simon Strasser Dec 31 '12 at 15:08
  • Another thing: It's not just about request/minute but also about continuity of requests and also about mass requests at unusual times. – Simon Strasser Dec 31 '12 at 15:10