Prevent my site from being copied

Question

Is it possible to protect my site from HTTrack Website Copier or any similar program?
Without setting a max number of HTTP request from users.

Answer 1

No, there's no way to do it. Without setting connection parameter limits, there's even no way to make it relatively difficult. If a legitimate user can access your website, they can copy its contents, and if they can do it normally with a browser, then they can script it.

You might setup User-Agent restrictions, cookie validation, maximum connections, and many other techniques, but none will stop somebody determined to copy your website.

Answer 2

Protect the part of the site you want to protect with a username and password. Then only assign a username and password to people who sign an NDA, (or similar) that says they won't extract or copy information from your site.

Another trick is to make all your content load from AJAX... and make the AJAX data URL load from paths that change (such as ~/todays-date) and sync that with javascript. Then even if someone were to download your content, the data would be out of date within 24 hours.

Even then, nothing will prevent a determined skilled attacker from getting an offline copy, you can just make it harder so it's not worthwhile.

Answer 3

As has @Adnan already pointed out in his answer, there is really no way of stopping a determined person from copying snapshots of your website. I used the word snapshots here, because that's what such content scrapers (or harvesters) are really copying. They don't (or at least shouldn't) have access to your backend where your website contents are actually generated and displayed to the end user, so the best they can do is copy its output, one that you can generate in such a way to change in time or adjust according to its intended recipient (DRM schemes, watermarking,...), as has @makerofthings7 pointed out in his answer.

So this much about what's already been answered. But there is one thing about this threat that I feel haven't yet been well covered in mentioned answer. Namely, most of such content scraping is done by opportunistic and automated web crawlers, and we see targeted attacks a lot rarer. Well, at least in numbers - bear with me.

These automated crawlers can actually be blacklisted quite effectively through the use of various WAFs (some might even use honeypots to determine the threats in heuristic ways) that keep updated database of blacklisted domains (CBLs or Community Ban Lists, DBLs or Domain Block Lists, DNSBLs or DNS-based Blackhole Lists,...) where such automated content scrapers are operating from. These WAFs will deny or grant access to your content serving web application based on three main approaches:

• Deterministic blacklisting: These are detections based on characteristics of web requests that content scrapers will make. Some of them are: Request originating IP address, Reverse DNS resolved remote hostname, Forward-confirmed reverse DNS lookup (see explanation in one of my questions here), User agent string, Request URL (your web application could for example hide a honeytrap URL address that a content scraper might follow in one of its responses, after it determines the request didn't come from a whitelisted address such as legitimate search engine crawlers / spiders)... and other fingerprint information associated with automated web requests.

• Heuristic blacklisting: This is a way to determine a threat either by weighting parameters of a single web request described in the deterministic approach (anti-spam filters use a similar approach based on calculating Bayesian probability), or by analyzing multiple web requests, such as: Request rate, Request order, Number of illegal requests,... that might help determine, if the request comes from a real and intended user, or some automated crawler.

• External DNSBL/CBL/DBLs: I've already mentioned relying on external DNSBL/CBL/DBLs (e.g. Project Honey Pot, Spamhaus, UCEPROTECT,...), most of which are actually a lot more useful than merely keeping track of spammers and spambot infected hosts, and keep a type of offense (e.g. forum spammer, crawl rate abuse,) on top of IP addresses, hostnames, CIDR ranges,... in blacklists they publish as well. Some WAFs will come with the ability to connect to these databases, saving you the trouble of being targeted by the same actor that might have been already blacklisted for same detected activity on another web server.

Now, one thing needs to be said quite clearly - none of these methods can be considered bulletproof! They will remove the majority of offending web requests, which is valuable on its own and will let you focus better on those harder to detect offenders that somehow bypassed your protections.

There are of course countless techniques for both automated crawlers / content scrapers detection (and their own countermeasures - detection avoidance techniques) that I won't describe here, nor list all possible WAFs and their capabilities, not wanting test your patience or reach limits of the purpose of this Q&A. If you'd like to read more on what techniques can be employed to thwart against such unwanted visitors, then I recommend reading through the documentation on the OWASP Stinger and OWASP AppSensor projects.

---

Edit to add: Suggestions from HTTrack authors can be read in the HTTrack Website Copier FAQ: How to limit network abuse - Abuse FAQ for webmasters document, and the reasons why a single deterministic method of detection won't work (short of blacklisting offending IP addresses after the fact or through experience of other honeynets), if the adversary is set to obfuscate spider's user agent string by setting it to any of the many user agent strings of real and legitimate web browsers, and disrespect robots.txt directives, become rather apparent by glimpsing through the HTTrack Users Guide. To save you the bother of reading it, HTTrack includes simple configuration and command line flags to make it work in stealth mode and appear just as benign as any other legitimate user to simpler detection techniques.

Answer 4

Everything the human user sees, he can record. As @Adnan points out, this is rather easy, and can be automated.

However, some sites still have some relative success at deterring mass slurping. Consider, for instance, Google Maps. Many people have, occasionally, tried to recover high-definition maps of large areas through scripting. Some have succeeded, but most were caught by Google's defences. It so happens that it is difficult to make an automatic downloader which acts, from the point of view of the server, as if it was under human control. Humans have all sorts of latencies and usage patterns which a shrewd sysadmin can notice and check on.

Similar tricks are done on, for instance, Stack Exchange. If you try to automate access to the site, you will soon be redirected to a page with a CAPTCHA.

Ultimately, this kind of security is not very satisfying because the defender and the attacker are on equal grounds: it is cunning against cunning. So, this is expensive: it requires thinking and maintenance. However, some sites do it nonetheless.

A generic way for attackers to defeat anti-automation safety measures is to "automate" the slurping with actual humans. Very cheap human workers can be hired in some countries.

Answer 5

I'd qualify what @Adnan says to add that while there's no way in general to prevent site leaching over time, a specific tool may exhibit behaviour that can be detected with some certainty once some amount of requests have been made. The order that URL's are accessed may be deterministic, such as depth first, breadth first, ascending or descending alphabetically, order that they appeared in the DOM and so on. The interval between requests may be a clue, whether the agent successfully executed some javascript code (NoScript and similar aside), client support for browser performance API, time spent between requests relative to page complexity, and whether or not there is a logical flow between requests. Unless a website leacher takes this into account, you may be in with a chance. User agent checking should not be effective as a good leacher would pretend to be a known bot, so unless you want to exclude Google and other search engines too, knowledge of the IP's that search engines use would be useful.

Answer 6

First of all, the only way you can prevent your site from being copied is actually never make it public to no one but you.

One way you could try to persuade people from doing it is with legal means, I'm not a lawyer so I don't know what steps you should take, if your content is original you could restrict the copyright or something similar.

I think that if you fear your site may get copied It has to be a really really really great web site.

Answer 7

Short answer, no, if the user loads a page, then the user can copy HTML by viewing the source.

If the website copier has a particular user agent, you can block that. See Stack Exchange for details.

Another solution might be to make a Flash webpage; those are hard to copy by hand anyways.

Otherwise, I'd put everything into a directory that has restricted access that only server-side PHP scripts can retrieve. Then if the page is built with many includes (one for a nav bar, one for header, one for javascript, one for footer, one for body content) then make another directory of php files that read the protected directory with includes, then make an AJAX that dynamically loads those PHP files. It would make it hard for anything to copy it without rendering the JavaScript (though I don't know if that would stop the software or an individual with a live code inspection tool.

Or you can put some kind of human-verification on your site so a protected PHP directory include isn't called unless the user specifically clicks a non-link DOM object (like a line that says "enter here") that triggers the content to load.

Answer 8

Disclaimer: this is an evil answer. I do not condone any of the following.

---

Modern browsers are capable of generic (Turing-complete) computation, through Javascript and possibly other means. Even their basic HTML + CSS rendering engines are incredibly elaborate pieces of software, capable of displaying (or hiding) content in a variety of ways. If that wasn't enough, all modern browsers make graphic primitives available, for example through SVG and Canvas, and allow downloading custom fonts to render text with.

If you put all this together, and some more, you will find there are a number of layers of execution between the site's source code and the pixels that make up the letters and words that the user can read.

All these layers of execution can be obfuscated and/or exploited.

For example, you can generate markup that has little or no resemblance to the graphic output, to make looking at the HTML source of your website an exercise in futility. You could use one HTML tag per letter, reordering them with a creative use of float: and position:, hiding some of them with complex, generated CSS rules, and adding some more that weren't there, with CSS-generated content.

You can create a font that uses a custom mapping between character codes and glyphs, so that copy and pasting your content would yield utter garbage, or even swear words! You can split letters in two or more pieces and use Unicode combining characters to put them back together. You can do all this with a dynamic generator, creating a new random masterpiece of obfuscation for each HTTP request.

You can write a program that will create complex Javascript algorithms, that when run on the client will fill in some required pieces of the puzzle, so that without Javascript support and a decent amount of client CPU time, the markup alone would be useless. 50ms of modern CPU time are unnoticed by most and are enough to run some pretty wicked algorithms.

Bonus points if you try to scrape your own obfuscated website using a headless browser, in order to have a full CSS and Javascript stack. Then try to find ways (or heuristics) to tell a real browser from the headless one. Then put some traps into the generated Javascript code, so that when it falls into the headless browser case, the algorithm goes into an infinite loop, or crashes the browser, or generates profanity and seizure-inducing flashes on the screen.

These are off the top of my head, there are (countably) infinite other ways to f*** with people's computers.

Now be a good boy/girl and take your blue pill :-)

Answer 9

First of all, as others have said - anything that you can see you can copy, using various methods. It depends why you want to prevent your website being copied, but the most effective method would probably be to add watermarks so that everyone knows where its come from. Perhaps even a polite notice asking people not to copy your website wouldn't go a miss.

However, going back to your original question and how to stop software from copying website, I believe CloudFlare has a web application firewall. I certainly know that Acunetix Web Vulnerability Scanner won't scan a website that uses CloudFlare. It's a free solution and it should also help speed your website up.

There is now foolproof solution though and anything can be circumvented. The best thing you can do is use a combination of the answers here, depending on how badly you need/want to protect your website. The best advice though, is if you don't want it copied, don't let people have it.

Answer 10

Even AJAX with date parameters can be duplicated. I've scraped sites with heavy AJAX using GET/POST parameters. If I really need to emulate the browser I can just use selenium or something of that sort. I can always find a way to scrape a site if I really wanted to. Captcha is probably the most difficult thing to deal with. Even then there's the Captcha sniper and other modules to assist in these areas.

Answer 11

Look at this links you may get solution from this :)

How to stop HTTrack?

Use robots.txt to prevent website from being ripped?

OR

The simplest way is to identify the browser id who is browsing your page , if it is htttrack block it, ( you need to configure your server or use ur programming skill to load the different page accordingly )

Thanks..