25

The Google crawl bot uses the user agent "Googlebot", and I am wondering if this knowledge can be used maliciously.

Let's say in a random website, the Googlebot user agent is allowed to access an admin panel, could this be considered a vulnerability, or is this just simply impossible?

CodesInChaos
  • 11,854
  • 2
  • 40
  • 50
Xavier
  • 351
  • 3
  • 4
  • 10
    One thing that might be interesting is where paywalled content-providers want to be indexed. You might assume that they rely on more than the user agent but I wouldn't bet on it. – Chris H Feb 23 '16 at 10:14
  • 44
    I find that some news sites, and Quora especially, are a lot less pushy about forcing me to sign up when I am Googlebot. I use [user agent switcher](https://chrome.google.com/webstore/detail/user-agent-switcher-for-c/djflhoibgkdhkhhcedjiklpkjnoahfmg) to toggle my UA between some common values, like Googlebot, the FB crawler, etc. It's kind of funny, and a little sad, that I have to impersonate a bot to get decent service. – Dan Ross Feb 23 '16 at 12:21
  • 3
    A user agent string isn't and shouldn't be used practically as an authentication method without some other means of authentication. – Sheerforce Feb 23 '16 at 02:16
  • @DanRoss And to think I've been entering https://www.glassdoor.com/Reviews/Lockheed-Martin-Reviews-E404.htm and https://www.glassdoor.com/Reviews/Lockheed-Martin-Reviews-E404_P2.htm , etc..., into Google and clicking the links like a caveman. – MonkeyZeus Feb 23 '16 at 18:15
  • 3
    @DanRoss Hehe, now that I've posted my comment I see that glassdoor.com removes the paywall for links from this site as well :-) – MonkeyZeus Feb 23 '16 at 18:17

7 Answers7

34

Yes. You should research what exactly goes in an HTTP request.

The user agent is easily changed as it is set by the client. There are many tools that allow you to change your user agent for your browser. If you are using curl, you can also just craft your own HTTP request and set the user agent to whatever you like.

If a web application depends on a user agent for security purposes, this is 100% vulnerable.

d1str0
  • 2,348
  • 14
  • 24
  • You answered my question of is it possible to modify it, but you didn't answer the question of can it be used maliciously. – Xavier Feb 23 '16 at 02:18
  • If a site is depending on it for any aspect of it's security then yes, it's vulnerable. – d1str0 Feb 23 '16 at 02:19
  • 1
    Okay, I understand what you mean now. So you are capable of using it as a authentication method, but it's not a recommended method. Thanks! – Xavier Feb 23 '16 at 02:21
  • 6
    @Xavier you should never use any data provided by the client for authentication. – Neil Smithline Feb 23 '16 at 02:29
  • 18
    @NeilSmithline that's not a very helpful comment. You always need some sort of client input for authentication. – d1str0 Feb 23 '16 at 02:30
  • @NeilSmithline I feel your statement is invalid, due to the fact information from the client will need to be grabbed. This consist of IP, browser, and so forth. – Xavier Feb 23 '16 at 02:32
  • 11
    By "data", Neil obviously meant information such as user agent, OS, etc, not fields sent by the client such as username/password (even those need to be validated but that's a different matter). He is absolutely right that you cannot trust this information (again, user agent, OS) to authenticate a user, since all of these can be spoofed. – Simon Feb 23 '16 at 02:54
  • @Xavier note that your question's title is *Is it possible to …?*, you didn't explicitly asked *how*. – A.L Feb 24 '16 at 16:32
20

Yes, a modified user agent can be used for malicious purposes. However it is unlikely that your scenario of the googlebot UA being used for privileged access.

How is UA more likely to be used? If your application parses the user agent to take some action, and you do not properly sanitize that input, Cross Site Scripting will be a very likely outcome.

In a similar vein if the user agent is parsed by some sort of server side scripting engine remote code execution could also be possible. Shellshock was a great example of that.

KDEx
  • 4,981
  • 2
  • 20
  • 34
  • 13
    Several paywalled sites allow the Googlebot to index their pages, then ask human visitors for payment. – Simon Richter Feb 23 '16 at 10:52
  • 1
    Back in my very (very) early days I wrote code to display web server logs as HTML. Did I properly sanitise the user agent before displaying it? I did not. Fortunately it was me who first developed and deployed a javascript injection attack against my own code, and not some enterprising bad guy. – Steve Jessop Feb 23 '16 at 12:39
  • @SimonRichter That's interesting. Wow. – Mark Buffalo Feb 23 '16 at 14:52
  • I don't think the UA string can be used for XSS, as it can't be set by malicious code (that does not already control the victims browser). – Bergi Feb 23 '16 at 16:27
  • 1
    @Bergi I guess I should clarify. In this case it's _stored_ cross site scripting. The UA is logged by the app and displayed in a separate panel back to the user. That is where the XSS is, and often may occur. – KDEx Feb 23 '16 at 16:42
  • Ah, you mean to another user (like in an admin statistics panel), I see. – Bergi Feb 23 '16 at 17:00
  • @SimonRichter - re those paywalled sites: sure, but the ones with any sense would use more than just the UA string to determine whether the visitor is actually the Google bot. – Simba Feb 24 '16 at 17:05
13

A recent example can be found in a Joomla exploit. Joomla stored the user agent in the session, and this allowed for a use-after-free vulnerability in PHP to be exploited.

Sjoerd
  • 28,707
  • 12
  • 74
  • 102
  • Another example: [an XSS in an CMS](http://www.sjoerdlangkemper.nl/2016/05/02/xss-in-useragent-header-in-bolt-cms/) that shows the user agent in the administrator interface. – Sjoerd May 02 '16 at 12:18
1

Yes, it's possible that it can be utilized for malicious purposes depending on how the application is written.

User-agents should never be used for any form of authentication and poses a huge risk from a security perspective.

moarsauce
  • 36
  • 1
1

Another example which User-Agent strings are also used, is in SIP. While your Blacklists/whitelists are often defined based on User-Agent strings.

For example the security scanning tool sipvicious uses the User-Agent (friendly-scanner). This User-Agent can be blacklisted so requests containing "friendly-scanner" are rejected, or better yet receive no response.

It is also a method to whitelist, an example would be in a IP-PBX scenario you can define the User-Agent of the SIP Phones/endpoints you are using and only allow their registration requests to be processed.

In this cases someone spoofing the User-Agent could use scanning/attack tools to circumvent the first layer of security to hopefully retrieve more information about the system(s) that reside in the SIP network or execute an exploit.

Stephen Gelardi
  • 111
  • 1
0

In addition to the answers above, an attacker can also launch SQLi attacks against a target website by supplying malicious input via the value for the User agent. For example:

User-Agent: NoAgent"; DROP TABLE users

More on this at: http://sechow.com/bricks/docs/content-page-4.html

https://security.stackexchange.com/a/1334

Also, example of how the user agent can be modified using Curl:

curl -A "Googlebot/2.1" http://example.com

Community
  • 1
racec0ndition
  • 581
  • 4
  • 10
0

While other answers detail if it is modifiable (you can), I'd like to talk about the malicious part of it.

You should never rely on user-agents alone.

Some D-link routers reportedly had a backdoor for their admin panel, which allowed everyone who set their user-agents to xmlset_roodkcableoj28840ybtide to login without password. You can read a blog post about it here.

ave
  • 191
  • 2
  • 11