93

Say, I have on my server a page or folder which I want to be secret.

example.com/fdsafdsafdsfdsfdsafdrewrew.html

or

 example.com/fdsafdsafdsfdsfdsafdrewrewaa34532543432/admin/index.html

If the secret part of the path is quite long, can I assume that it's safe to have such a secret page or area, and it'll be hard to guess or brute force it?

What are the issues with this approach in general?

Note that I'm not asking how to do this right, but what could the issues with this approach be, if any.

Kargari
  • 911
  • 1
  • 6
  • 5
  • 2
    provided your machines are secure (https, good wifi, etc), and you never publically link to the sercret URL, then a long static URL would be very hard for an outsider to guess. – dandavis Jun 19 '18 at 06:24
  • 43
    oh, and you've turned off directory listings and "smart error" pages that suggest paths... – dandavis Jun 19 '18 at 06:30
  • 62
    Note that this practice is considered "good enough" by some well-established web services such as Google Docs (which has a share-by-link feature) and Overleaf (a collaborative online Latex editor). – Federico Poloni Jun 19 '18 at 07:50
  • 9
    @FedericoPoloni: Google is likely going to track bulk requests and eventually block them. Without that in place, it becomes _more_ likely to succeed by bruteforcing (relative to the url length of course). – Flater Jun 19 '18 at 07:57
  • 2
    Some websites do this, for example for password reset links. Your safest bet, if you decide going down this road, is making the secret a GET parameter (so `index.html?secret=hjasdhasdhasdasd`) to avoid caching and robots accidentally stumbling upon your link, and making it **temporary** – BgrWorker Jun 19 '18 at 14:22
  • 3
    @BgrWorker Tokens in password reset links should be used once only and discarded after that. What OP is asking is a fixed value that can be reused, it's not exactly the same thing – Mr. E Jun 19 '18 at 16:28
  • As long as the path was never made public, it would be pretty hard to guess, however this is security through obscurity, so there is nothing really security the URL. My advice would be to expand this a little bit and make the url hash dynamic, so everything index.html was hit, it would be on a different, but known path, easier to do in some frameworks than others. – RandomUs1r Jun 19 '18 at 21:16
  • if it's over https, it's not as bad as passing in a password, or something that can also be used for other purposes. – Rob Jun 19 '18 at 23:33
  • @FedericoPoloni Doesn't Google Docs allow access to be restricted to certain accounts, so others cannot access even if they guess the URL? And Overleaf V2 keeps link sharing off by default. – GoodDeeds Jun 20 '18 at 06:11
  • 4
    @GoodDeeds Share-by-link is one of the allowed sharing options on Google Docs. You can also decide to share to specific accounts, but that's a different thing. – Federico Poloni Jun 20 '18 at 06:12
  • This is basically security by obscurity which - by itself - is never sufficient. – CompuChip Jun 20 '18 at 08:33
  • 3
    @CompuChip No it's not. No more than a password is "security by obscurity" because not everyone knows the password. –  Jun 20 '18 at 14:20
  • @DrEval maybe I misunderstood the question then but what I read was that OP just wants to obscure the page by making the URL harder to guess rather than properly protect it e.g. using a password. – CompuChip Jun 21 '18 at 01:58
  • @CompuChip If the URL is secure enough, it "becomes" the password. It's still insecure, since this isn't a good way to transmit passwords, but it's not security through obscurity. – Brian McCutchon Jun 21 '18 at 04:08
  • If you have a robots.txt document that can give away some urls, though if they're meant to be secret chances are you never put them in robots.txt to begin with. – Shelby115 Jun 22 '18 at 13:05
  • If someone can look at webserver logs they could find all of those interesting URLs. – Salman A Jun 22 '18 at 17:07
  • Also related: [How unlikely is it that a Google Doc link is guessed?](https://security.stackexchange.com/questions/29449/how-unlikely-is-it-that-a-google-doc-link-is-guessed) – Ajedi32 Jun 22 '18 at 20:04
  • Visit ["security through obscurity is a bad idea"](https://stackoverflow.com/questions/533965/why-is-security-through-obscurity-a-bad-idea). – Second Person Shooter Jun 23 '18 at 04:41

8 Answers8

122

You are essentially asking if it is safe to pass secret parameters in a GET request. This is actually classified as a vulnerability. It is not feasible to brute force a sufficiently long pseudorandom string, assuming the server simply returns a static 404 response whenever an invalid path is specified, but there are numerous other security issues in practice that make this a dangerous technique:

  • Logging software often logs GET requests in plaintext, but not POST.

  • Using GET makes CSRF trivial* when processing active content.

  • Referer headers may leak secret values to other websites.

  • Browser history will retain secrets passed in GET requests.

Your second example has /admin/ in it. This implies to me that knowledge of this path alone would be sufficient to authenticate to the server in an administrator context. This is very insecure and should not be done anymore than /?password=hunter2, a major web security faux pas.

Instead, secrets should be sent in a POST request. If this is not possible to do or if your threat model is exclusively to prevent brute force of the URL (for example, password reset links which are promptly invalidated after they are used), then it should be safe if done carefully. I am not aware of any side-channel attacks that would provide a method to obtain the string faster than brute force.

* Avoiding parameterized GET requests does not prevent CSRF attacks, which is a common misconception as there are various ways to similarly abuse POST (fake forms, etc), but passing secrets in GET does make CSRF easier.

forest
  • 64,616
  • 20
  • 206
  • 257
  • 2
    *"[CSRF](https://www.owasp.org/index.php/Cross-Site_Request_Forgery_(CSRF)) becomes an issue. In fact, using only POST for secrets mitigates CSRF."* As explained in your linked OWASP article, this does not actually prevent CSRF. – Alexander O'Mara Jun 19 '18 at 05:30
  • I'd argue it really only makes it a little harder in some cases, since JavaScript can submit forms cross-origin without any user interaction. The edit is good though. – Alexander O'Mara Jun 19 '18 at 05:34
  • Yep. The only way to block it would be to reject external referrers. – Alexander O'Mara Jun 19 '18 at 05:37
  • `It is technically not possible to brute force such a random string,` --> why not? – Kargari Jun 19 '18 at 06:27
  • 3
    @Kargari Because the server will not tell you how close you have gotten to guessing the string. If you give it the wrong string, it will return 404 whether or not it's 1 character off or 50 characters off. As such, you would have to exhaustively search through all possible strings until you get _exactly_ right. Anyway, I probably should have said _not feasible_ rather than _not possible_. Edited it now. – forest Jun 19 '18 at 06:29
  • 9
    timing attacks may be possible though; i doubt webservers and OSs use fixed-time comparisons when checking for the existence of paths! – user371366 Jun 19 '18 at 07:14
  • 1
    @dn3s I am not aware of any practical side-channel attacks against popular HTTP servers. – forest Jun 19 '18 at 07:15
  • `the server will not tell you how close you have gotten to guessing the string. If you give it the wrong string, it will return 404 whether or not it's 1 character off or 50 characters off. ` --> of course. that's where brute-force comes in – Kargari Jun 19 '18 at 08:58
  • 4
    @Kargari Brute force is very, very slow. For a 30 character alphanumeric string (i.e. a-z, A-Z, and 0-9), you have 62^30 possible combinations which is _far_ beyond what is possible today. – forest Jun 19 '18 at 09:01
  • Maybe @dn3s means that if the server is comparing character by character it will stop at the first difference, allowing for timing attacks. But I think the time difference may be too small to measure. – beppe9000 Jun 19 '18 at 11:27
  • 4
    You can mitigate that timing attack by storing a hash of the random string. So the time would tell you how many characters of the hash match, but that's not correlated to how many characters of the original string match. – Barmar Jun 19 '18 at 17:19
  • it was a bit of an obscure thing for me to mention and i was unable to find examples of this from a quick search. i am curious now if this is a thing or not (or just generally, what kind of filesystem information can be extracted from a server using timing side-channels). I may have to make a question on this! – user371366 Jun 19 '18 at 18:00
  • That OWASP link is being misinterpreted, they're referring to information, a url hash isn't "information". It is however, security through obscurity. – RandomUs1r Jun 19 '18 at 21:11
  • As for the referer header, you can prevent it from being sent [by adding `rel="noreferrer"` to the link](https://stackoverflow.com/questions/5033300/stop-link-from-sending-referrer-to-destination). – Michał Perłakowski Jun 20 '18 at 16:23
  • If this obscure (not secret) path is ever accessed over http and not https, the URI will be visible and possibly logged on any intervening system. Also, if it is ever accessed on a shared computer, it may be recoverable in the browser's history. Asking the question is good, but that OP is asking what the issues are in the first place, implies that this probably isn't a good idea to do, in what ever form OP is planning on using it. – JesseM Jun 20 '18 at 17:39
  • 1
    Isn't this exactly how "magic links" work? – Christian Wattengård Jun 21 '18 at 12:29
  • Though not as applicable anymore, I think ActiveX vulnerabilities would allow the address bar to leak if you typed to go to that site directly from the malicious site. – mbomb007 Jun 21 '18 at 22:09
  • 1
    Source of `hunter2` pass: http://bash.org/?244321= – Stefan Dragnev Jun 21 '18 at 22:10
  • Per the [OWASP REST Security Cheat Sheet](https://www.owasp.org/index.php/REST_Security_Cheat_Sheet#Sensitive_information_in_HTTP_requests) you should not pass sensitive data in URLs but it's OK to do so in headers. – JimmyJames Jun 22 '18 at 16:00
  • So, why does Google use this? – enkryptor Jun 23 '18 at 11:26
  • @enkryptor Google does not do this. Sending search queries via GET is not the same as sending your password via GET. – forest Jun 23 '18 at 19:20
  • @forest who is talking about sending passwords? OP asks about kind of a secret link, "a secret page" he said. For me, it was neither about sending password to his account, nor about gaining privileges somewhere else. It is just a secret link. – enkryptor Jun 23 '18 at 19:45
  • @enkryptor Then what do you mean by Google doing this? – forest Jun 23 '18 at 19:46
  • @forest Google's link sharing allows you to provide access to a private document with link. The link itself becomes a "password" for this document, until the document owner allows it – enkryptor Jun 23 '18 at 20:01
34

This is a common approach to share public things restricted to the ones who know the URL. An example is Google Docs:

enter image description here

The second option, "Anyone with the link", creates a link similar to yours. Same for Google Photos, Dropbox, ...

  • The advantage is that the diffusion of the content is somewhat limited.
  • The drawback is that this somewhat depends with whom you share the link, where it is published, etc.

One thing you should consider is the possibility to invalidate it easily (by changing/regenerating the link to your data)

WoJ
  • 8,957
  • 2
  • 32
  • 51
  • 1
    This is also a common mechanism for "unsubscribe" links. – Micah Epps Jun 20 '18 at 18:48
  • 3
    This answer cites Google Docs, but (rightly) mentions that Docs doesn't create the link _until the owner decides to share it_. That's different than just having a link from the moment the content is created, as in OP's case. I cannot brute force an arbitrary user's private docs, only the ones for which they've intentionally made public links. – Knetic Jun 22 '18 at 05:22
  • I would also add that Google has _vast_ knowledge about virtually everyone (or at least every browser; having or not having a Google account doesn't matter, everyone with a browser has an internal account with Google) and they may utilize it in the background when granting or denying access to a document available via the unique link. Question is whether the OP has similar data and tools at their disposal. – Pavel Jun 22 '18 at 07:17
  • @Pavel: I am not sure I understand. What do you mean by "*everyone with a browser has an internal account with Google*"? And by *they may utilize it in the background when granting or denying access to a document available via the unique link*? – WoJ Jun 22 '18 at 07:23
  • 1
    @WoJ Google actively tracks browsers and their users (Analytics, Tag Managers, APIs, Fonts, reCaptcha, 8.8.8.8 etc are used very heavily throughout the web), so that when your browser requests the document via the unique URI, Google already knows more than you'd like about you (of course tracking the life of a unique Docs link is a part of the same story). That makes it easy to challenge or even filter out foul players. – Pavel Jun 22 '18 at 09:44
  • @Pavel: ah yes, now I understand what you mean. It is true, and the mechanisms used to fingerprint users (not only by Google but by others as well) are quite powerful. The mere fact that some sites protected by a TOTP/HOTP mechanism (the F2A used by Google Authenticator for instance) can recognize an incoming device despite the cookies having been cleaned up is impressive. – WoJ Jun 23 '18 at 13:46
20

Bad idea. A number of times I have seen a "secret" URL very quickly getting search engine crawler hits, and then discoverable by web search. Once I even saw someone set up a copy of a reputable website in a subfolder of his domain, share it with one person, and soon he was emailed a notice warning him that his domain may have been compromised for phishing purposes.

How did this happen? In the latter case, the web browser had a built in anti-phishing feature which sent visited URLs to fraud detection services. In the other cases, perhaps the browser was collecting browsing data and sending it to a search engine to collect user habits.

If you do this, make sure your robots.txt (or headers/meta tags) is set up to tell search engines not to index the content.

The internet is a wonderful way of bringing the world closer together, but unfortunately it gives everyone potentially permanent access to anything you happen to sneeze out.

Artelius
  • 588
  • 2
  • 4
  • 31
    `engines not to index the content.` --> not to index my secret page by adding its url into robots.txt? – Kargari Jun 19 '18 at 15:58
  • 34
    **Never put any secrets in `robots.txt`!** The best way to make sure **everyone** finds out about your secret page is adding it to the `robots.txt`, because then all that needs to be done to discover the secret is to look at the `robots.txt`. – Moshe Katz Jun 19 '18 at 20:35
  • 31
    @Kargari There's no reason to put the url of the actual *secret page* into robots.txt. A robots file is perfectly capable of disallowing entire directory hierarchies. If your secret is in /nocrawl/page/sdghskdfjgneowsvnoiernow.htm then a "Disallow: /nocrawl" directive will apply to it. – jmbpiano Jun 19 '18 at 22:21
  • but I don't have "'/nocrawl" directory in my path and don't want to add it to my routes only to serve the purpose of robtots.txt! – Kargari Jun 20 '18 at 02:12
  • @MosheKatz I will put it in there – Kargari Jun 20 '18 at 02:14
  • 2
    Crawlers don't have to obey `robots.txt`. I suspect there are programs that do the opposite: only inspect things that have been 'hidden' by `robots.txt`. – Pharap Jun 20 '18 at 08:09
  • 6
    @Kargari "/nocrawl" is an example not a prescription. Please [read up](http://www.robotstxt.org/robotstxt.html) on how robots.txt files work before attempting to use one. – jmbpiano Jun 20 '18 at 14:03
  • 1
    @Pharap A crawler won't get to there unless the URL is visible in a public space, at which point the URL is compromised anyway. This answer is suggesting that certain tools endusers might use may well do 'helpful' things that expose the URL in public or semi-public places. – Cubic Jun 20 '18 at 14:56
  • 4
    *"the web browser had a built in anti-phishing feature which sent visited URLs to fraud detection services"*... wait, what? So that fraud-detection feature can see every Google doc or whatever you ever had "shared by link"? – user541686 Jun 20 '18 at 22:41
  • iirc robots.txt syntax allows you to deny all paths/files and then specifically allow paths and files. – ivanivan Jun 21 '18 at 22:00
  • 1
    Somewhat related: I once sent a _secret_ URL to someone via email, only to find that the URL was accessed before they opened the email. Turns out it was accessed by Yahoo Mail's include "link previews in email" feature at the time I was composing the email. The user agent had an "about..." URL, and that page recommended using robots.txt to block this user agent. – Salman A Jun 22 '18 at 17:16
9

If the secret part of the path is quite long [...] it'll be hard to guess or brute force it?

Yes. An attacker would have to guess the whole thing to discover it. Since there are many possibilities it would take an infeasible amount of time.

What are the issues with this approach in general?

The problem with this is that URLs are not considered secret, so they will be stored in the browser, in logs and by any intermediate proxies. That way, your "secret" URL may be exposed.

Using GET makes CSRF trivial when processing active content.

No. In a CSRF attack, the attacker forges a specific request. When using a secret in the URL, the attacker does not even know the correct URL to forge the request to. This means that CSRF is impossible as long as the URL is secret.

Sjoerd
  • 28,707
  • 12
  • 74
  • 102
4

As others have stated, this isn't a good idea. Such "secret" links that are used to unsubscribe or similar one-off purposes are typically relatively short-lived, and of no use once they've been used once.

My initial thought when I read this question was that the url would not stay secret for long. I thought that perhaps Chrome (or any other modern web browser) would use the data from the address line to initialize a crawl. It turns out that they don't. However, the researchers discovered that plugins might still trigger a crawl:

The results are pretty simple: Googlebot never came to visit either page in the test.

As it turns out, two people in the test did not actually disable their extensions, and this resulted in visits from Open Site Explorer (someone had the Mozbar installed and enabled) and Yandex (due to a different person, though I’m not clear on what extension they had installed and enabled).

This means that once a link is used, it may be compromised by browser extensions. And then there's no need for brute forcing anything.

What is the purpose behind making these links? What are you, OP, trying to achieve? I'm certain that there are other ways to get there...

Thomas
  • 41
  • 1
2

It'd be hard to guess/bruteforce but other ways to obtain the paths may be possible

For example, the url may be indexed by services such as google, bing, etc. This would make your "secret" url appear when an user searches your page in google. It can be solved configuring the robots.txt file, but remember that indexers may ignore it

Links in the application may redirect to the hidden path

In addition, machines in the same network as the user accessing the "secret" page or the web server can see the url, also every intermediary proxy and the user's ISP (Or VPN if he uses one)

Finally, the url may be persisted in the browser's cache and/or history and in the logs on the webserver and proxies

Mr. E
  • 1,954
  • 9
  • 18
  • 3
    **Never put any secrets in `robots.txt`!** The best way to make sure **everyone** finds out about your secret page is adding it to the `robots.txt`, because then all that needs to be done to discover the secret is to look at the `robots.txt`. – Moshe Katz Jun 19 '18 at 20:36
  • 3
    @MosheKatz I didn't say anything about adding the secret to `robots.txt`, but configuring it so your pages don't get indexed. Something like `Disallow: /` – Mr. E Jun 19 '18 at 20:46
  • 7
    @MrE you may know that, but people reading your answer may not understand that. – Moshe Katz Jun 19 '18 at 21:02
2

Can secret GET requests be brute forced?

Yes, they can. As much as any type of request without proper security measures.

If the secret part of the path is quite long, can I assume that it's safe to have such a secret page or area, and it'll be hard to guess or brute force it?

No, it's not safe to have such secret page or area. Yes, it will be hard to guess or brute force it.

What are the issues with this approach in general?

Unlike POST requests, GET requests can be easily found in the web browser history.

Gillian
  • 492
  • 1
  • 3
  • 13
0

I think that it will better to implement an automatic or manual (so you'll decide who can access to the page) OTP One Time Password to send via email.

In a manual scenario:
- you receive the request from email@example.com
- you grant access to email@example.com
- the script create the link with the OTP
- the link will be mailed to email@example.com
- when the user email@example.com will visit the page the OPT will be flagged and nobody else can reuse that link

of course this system require a database where store the authorized email, OPT and flag field

ViDiVe
  • 1
  • 1