5

When would a human outperform a scripting tool when testing for vulnerabilities? For example, would a human ever find something that SQLmap wouldn't? In what cases would it be preferable/are there any examples?

For example consider the url: 

"fakeurl.com/vuln.php?id=4"

A human could try:

"fakeurl.com/vuln.php?id='"

and see if an error message pops up, but in that time he could just run

sqlmap -u fakeurl.com/vuln.php?id=4 --batch --dump-all

and find everything so much faster.

I know a lot of people like to look down on script kiddies, but for practical, professional purposes it's best to just do things fast and properly. It seems to me that working by hand is less efficient and more prone to errors. Can someone give me a case (technical or a historic example) where a human found something a tool wouldn't have?

user3364161
  • 63
  • 3
  • 1
    There are actually going to be a very, very long list. Automated tools are limited by the imagination of the devs at the time of coding, and they have to account for general cases. A human can adjust to the target at hand. Sure, automated tools can find stuff *too*, which makes them useful. But they will not find *everything*. – schroeder Oct 31 '16 at 22:07
  • It seems like the answer to your question is in the very tag you gave the question. Tools are just that, tools ... humans made the tools to automate things faster than they can by hand ... but it took the humans intuition to create the tool and it will take the humans intuition to defeat the tool, and again human intuition to improve the tool. The problem with script kiddies is that they are using the tool w/ out understanding it and thus do not have the intuition to use it optimally or improve it when needed. – CaffeineAddiction Nov 01 '16 at 15:24

2 Answers2

9

Most access control / authorization flaws would never be found by a (generic) tool, because it does not have the understanding of what is supposed to be accessible and what not. (Having said that, experienced pentesters probably know that quite a lot of applications also don't have this documented...) So that is an example of a whole class of problems.

Any logic flaw (for example a user being able to create another user with more privileges by design) would also not be picked up by an automated tool.

Any chain of vulnerabilities would not be correlated, like for instance how to use a low risk info leak, together with a medium risk DOM XSS together with another low risk vulnerability to change a user's password to anything the attacker wants (this is an actual example I have seen).

Testing DOM XSS with a plain (usual) scanner like most of those commercial tools is quite difficult as they don't have a Javascript runtime, so they will miss most of it.

Also for things that are testable, a tool may have several patterns to try, but will probably miss more complex cases. For example what if an application had a blacklist filter for XSS that explicitly blocks alert(1) and all of the attack vectors from the tool had that as the payload? Getting through a blacklist filter is almost always possible, but is very hard for an automated tool.

Or consider DoS. How would an automated tool find that?

For a final example, what about buffer overflow in an uploaded and processed file, say a user uploads an image which gets resized by a server side library, vulnerable to bof. How could the automated tool know that's the case and how would it create an exploit for it?

These are just a bunch of examples, I'm sure others will cite a lot more.

In short,

  • there are whole classes of vulnerabilities that cannot be tested for in the generic case

  • even for vulnerabilities that are automatically testable, it is pretty much impossible to write comprehensive tests (a set of tests that finds all instances of the vulnerability).

Of course with all of this in mind, I think there is nothing wrong with using tools to make several things quicker. However, any result from a tool needs to be reproduced by the tester, and also he must be aware of the limitations of the tool to be able to augment results with more creative attacks.

Gabor Lengyel
  • 1,163
  • 7
  • 11
  • Those are a lot of good examples, and I can see how humans could do stuff better than tools frequently (e.g. almost all social engineering, logic errors). I even understand how working through SQLi by hand can be beneficial to practice and learn fundamentals. What I meant was in cases where it's simply one exploit, just a pure SQLi, what could a human do that a computer couldn't? Blacklisted cases to prevent certain tools, what else? – user3364161 Oct 31 '16 at 01:27
  • 1
    A tool will always just have a list of test cases. That may be very smart, but will never cover *all* possible ways an application may be vulnerable to say SQLi. Sticking with the SQLi example, I'm not saying a tool is useless, most of the times they are pretty good for SQLi actually. But especially in a blind case, a human's understanding of things like the underlying architecture may help a lot in finding an actual vulnerability. – Gabor Lengyel Oct 31 '16 at 01:35
  • For a more specific example, imagine a web application that takes XML as its input and several fields make their way into a vulnerable SQL statement that's run in a background job. A generic tool has like 0 chance to find this, while for a human it can be obvious that there's something to look for. – Gabor Lengyel Oct 31 '16 at 01:37
  • I'm working on an SQLi practice challenge. It's super contrived: a PHP application with just a "username" and "password" box as a post form. Somewhere on the site there's a hidden flag, but the error message isn't verbose (just "Incorrect Login") and no attack vectors other than the text boxes. I've tried modifying the URL and text boxes, but with no error or feedback I have no idea where to go. I ran sqlmap on it which was also unsuccessful, and the only remaining thing I can think of is running a time-based SQLi. Is there anything that I can do as a human that a machine can't? – user3364161 Oct 31 '16 at 01:48
  • 1
    Not in the sense that you can technically make the very same requests that a tool can. What you have and the tool doesn't is creativity. :) (What if the SQLi is on a cookie value or another request header? What if it's on one of the fields when the other has the correct value, like in your case on the password when the username is 'admin'? Things like that. A tool would never find the latter.) – Gabor Lengyel Oct 31 '16 at 01:55
  • So what are some things that you'd recommend trying on a blind SQLi? I've checked the cookies (there are none), I've tried the basic "' or 1=1;--" stuff, I've run sqlmap at the highest level/risk, and found nothing. My creativity is failing me, what else is worth looking for? – user3364161 Oct 31 '16 at 02:16
  • Maybe you need to encode the input data, if it is an PHP application, then it could connect to a MySQL DB, so you can try with: `'--%0Aor--%0A1=1;--`; this is a kind of encoding to bypass black lists. – hmrojas.p Nov 01 '16 at 16:19
1

I just want to complement the answer above, vulnerabilities like SQL Injection and XSS can be detected by systematic processes (not always), a tool would be enough for this work, but as @lengyelg said, there are complex cases which can't be detected by a tool ease, for example: a Blind SQL Injection where the injection could be after order by or limit, the application is behind a WAF and the response from the application is a JSON data structure; this kind of vulnerability needs time to be analyzed, If you run a tool, it might not able to detect the vulnerability, I don't want to say that the tools aren't able to detect them, but a tool needs knowledge to be configured correctly. The best way to get this knowledge is doing it yourself, knowing the application, a correct response, an incorrect response, an error from the application, etc; when you can understand the behavior of the application and each of its components very well, you can build an good exploit and configure a tool to do the work faster and efficient. A tool help us to do our work, but in my oppinion, it should never do our work completely. Then I think a tool needs the human knowledge, a hammer needs the right strength and the right direction to do a good work.

hmrojas.p
  • 1,049
  • 1
  • 8
  • 16
  • The reason I asked the question to begin with is because I'm working on a blind SQLi challenge and I don't really know where to go. I don't know how to proceed without output from the exploitable application, and none of the tools can find any information for me to dump and look through. What I'm really looking for at this point is suggestions of what a human could try when the tools fail. Is there a certain way that I can configure SQLmap to work better that I should know about? I can't even confirm a true case vs. a false case, the website just outputs "Wrong Password" no matter what. – user3364161 Nov 03 '16 at 05:29
  • Well you should analyze the complete reponse from the application, indentify the HTTP code, the HTML content, the lenght header, etc; they could show you some difference between a true sentence and a false sentence. Now, if this doesn't work, then it could be a Blind SQL Injection based on time. – hmrojas.p Nov 03 '16 at 15:02