How can I prevent client software scraping data from my website?
Some examples of my URLs are:
It's very easy to run a loop and request the server with a changed query string. What are the security measures I can take? The information on my site is published weekly (huge data entry cost), and I don't want someone to run a script and get all of it in a few minutes.
You can't really prevent it if the data is publicly available.
But that doesn't mean that you have to make it extra easy to scrape the data either.
Preventing Enumeration
By exposing internal, ordered ids you make it extra easy to scrape all products.
If you changed it to either product name or a random id, an attacker couldn't retrieve all the data with a simple loop.
Throttling Requests
You could limit the amount of requests a user can make. This isn't all that easy though, because you can't really limit by IP address (you would restrict users using the same IP address , and attackers can just change their IP address). Here is a question about identifying users with some alternative ideas.
"Honeypot"
You could create fake products, which you never link to (or only in links hidden via CSS).
When someone views such a product, ban them.
Alternatively, you could add quite a lot of these products, and NOT ban a scraper, but just let them keep the wrong data, making their data less accurate (this may or may not make sense in your case).
Obscure Data
You could try to make it harder for a scraper to use your data. This may impact your users and may be quite a bit of work on your part (and as with all approaches, a determined attacker can still get the data):
Limit Access
You could put the content behind a login, and ban users that scrape data (probably not a good idea in your case, as you do want users that don't have an account to see products).
The Law
Everyone is free to scrape the data on your website (probably; it may depend on your jurisdiction). But re-publishing is likely in violation of the law, so you could sue them.
Scrape Artist here.
There is no reliable way to do it. You can only make it harder. The harder you make it, the harder you make it for legitimate users. I write web scrapers for fun, and not only have I bypassed all of the above ideas by tim, but I can do it quickly.
From my perspective, anything you introduce to stop me, if your information is valuable enough, I will find a way around it. And I will do all of this faster than you can fix it. So it's a waste of your precious time.
Throttling requests: bans legitimate users who look around. Doesn't matter: I will test to see how many requests are allowed in a certain period of time using a VPN. If you ban me after 4 attempts, I will update my scraper to use my distributed proxy list to get 4 items per attempt. This is super easy, as every single connection asks me if I want a proxy. Simple example:
for (int i = 0; i < numAttemptsRequired; i++)
{
using (WebClient wc = new WebClient())
{
if (i % 4 == 0)
{
wc.Proxy = proxyList[curProxy];
curProxy++;
}
}
}
I can also add a simple method to make sure it doesn't happen more than a few times per second, at the same speed as a regular user.
"Honeypot": bans legitimate users who are looking around and will likely interfere with the user experience.
And in all of this, I can impersonate a legitimate user by sending fake user-agents, etc., based on real values.
Not really. Like I said, you can only make it harder to access. In the end, the only people getting hurt will be your legitimate users. Ask yourself this, "How much time would I spend on this fix, and how will it affect my customers? What happens if someone finds a way around this quickly?"
If you try to make your website too difficult to access, you may even end up introducing security holes, which would make it even easier for malicious visitors to exploit your website and dump all of your information without needing to scrape it. Even worse, your customers could be affected negatively.
If you limit attempts and require authentication, this can slow down the aggregation of your main website, I can still find everything in search results/linked pages.
In the end, it doesn't matter. I'll still get around that with my proxy list. I'll still get all of your data much faster than a normal user could.
I believe that if you present a quality browsing experience to your users, and you have a good product, people will come to you even if others have the data as well. They know your website works, and they aren't frustrated with bugs, nor are they plagued with usability problems.
Here's an example: on amazon.com, I can aggregate almost all of their products very quickly by changing a few numbers. If I take that data, where does it get me? Even if I have the products, people will still be visiting Amazon.com, and not my knock-off website.
As well as the excellent points in Tim's answer, there are a couple more options
Complain to their ISP
If scraping your site is a violation of your terms & conditions, then if you complain to the ISPs of scrapers you have identified from their logs, they will generally tell them to stop doing it.
Live with it
Try to quantify the damage the scraping is doing to you. Compare that with the effort required to stop it. Is it really worth worrying about, or is it just an annoyance that is best ignored?
