How does Facebook avoid crawling over their site?

Question

I used Netsparker community edition and tried to find a vulnerability in my personal site. Then I thought let's try with Facebook. It didn't show a single webpage also because normally it shows links inside the website, whole site structure but here it didn't.

Why? What is the reason?

rook · Accepted Answer · 2012-03-30T18:05:32.633

9

Other than the user's privacy settings, Facebook doesn't inhibit spiders. My scans show they are not even using a Web Application Firewall to block requests. The load to scrub every HTTP request of the most popular website would be massive undertaking. The problem is that your scanner isn't equipped to interact with a mostly JavaScript client communicating to a REST backend. Netsparker is also unable to detect DOM Based XSS, which is more common on facebook.

edited Mar 30 '12 at 18:05

answered Mar 30 '12 at 15:42

rook

46,916
10
92
181

Good point. I just checked and Facebook actually works reasonably well with javascript turned off now but it certainly didn't last time I checked. Back then, if I remember correctly, you got nothing but a redirect to a page telling you to turn javascript on. – Ladadadada Mar 30 '12 at 16:40
Netsparker Professional does detect DOM XSS, but not every one – atdre Apr 02 '12 at 01:09
@atdre Yeah probably pretty shitty at doing it. DOM based XSS is a static analysis job, and even then these methods produce a lot of false positives. – rook Apr 02 '12 at 01:10
@ Rook: I would say that it is never a good idea to fully automate any difficult or complicated vulnerability class, and DOM XSS certainly fits into this category – atdre Apr 02 '12 at 01:15
2

@ Rook: OTOH, HP WI-RT 9.20 can provide the dynamic tester tool a WADL of the RESTful API so that parameters are executed properly, while at the same time providing all of the client-side submitted serialized/unserialized XML/JSON/etc that is populated into the data flow of the application. It can't exercise all control flow, but SecurityScope will provide cross-references to the source code sections if HP AWB/SSC or sourceanalyzer are used along with the parameter/event-logging FPR that is generated on the server. You can also control these programmtically through various web attack APIs. – atdre Apr 02 '12 at 01:20
@atdre I agree. I think that a mix of run time analysis and SCA is a good approach to the problem. I also agree that full automation is never a good approach, one problem being that not every API has a WADL or WSDL. – rook Apr 02 '12 at 14:12

score 5 · Answer 2 · answered Mar 30 '12 at 07:54

If you aren't logged in to Facebook, the only page you will get is the login/signup page.

They're likely also running an IDS/IPS that detects and blocks vulnerability scans. (Although I do vaguely remember something about Facebook offering bounties for vulnerabilities found in their website, anything that a free scanner could find will have already been found and fixed.)

Indiscriminate vulnerability scanning of third-party websites without their permission is not a good idea. It's somewhat impolite and has the potential to get you in trouble. The 3am-FBI-raid-and-20-years-in-prison kind of trouble.

Indeed Facebook have a bug bounty programme https://www.facebook.com/whitehat/bounty/ I'd agree though would be surprising if a scanner found anything of interest... — Rory McCune, Mar 30 '12 at 12:24

score 3 · Answer 3 · answered Apr 02 '12 at 01:13

I have discussed with previous lead developers of the Facebook portal this very subject, and they do indeed block crawling of their domains and applications, but this is usually on a case-by-case basis.

Often, captchas and other slowdown mechanisms are employed to prevent bots, scrapers, and crawlers from siphoning their content. Often times this is moreso because of your source IP, especially if it is a known closed/open proxy, a Tor node, on the FATF blacklist, or other distinguishing feature relevant to their operations.

It really is best not to mess with Facebook or similar large sites without permission because many curious people in the past have been slapped with very nasty lawsuits and/or waves of criminal justice.

How does Facebook avoid crawling over their site?

3 Answers3