31

I'm reading a white hat hacking book from a famous certification. They say the methodology for hacking a web server is:

  • information gathering (domain name, DNS, IP, etc.)
  • footprinting (ex: banner grabbing)
  • website mirroring
  • vulnerability scanning
  • session hijacking
  • password cracking

Apart from session hijacking and information gathering, I don't see why I would not just launch Acunetix Web App Scanner and/or Nessus to find all weaknesses.

What is the point of performing manual tests if you can automate them?

For instance, if the vulnerability scanner does not know how to find vulnerable cookies, and if I manually find a way to do session hijacking, I won't be able to train Acunetix of Nessus for that. Even if I did, I don't how beneficial it would be.

Please explain why I would not just let my tool do the hacking for me.

Toby Speight
  • 1,214
  • 9
  • 17
botanga
  • 437
  • 4
  • 5
  • 62
    Why bother learning a language if you have Google Translate? Because automated tools don't work as well as people. –  Feb 29 '20 at 13:25
  • 3
    @MechMK1 Hahaha. Nicely said. I think you've conviced me. This should be an answer – botanga Feb 29 '20 at 14:50
  • 2
    I had several projects pass all Nessus scans then asked a security consultant to take a look and 12 hours later he logged in as an admin account and changed my password - vulnerability scanner can't find all vulnerabilities simply because software does not have creativity – slebetman Mar 01 '20 at 03:55
  • 5
    A tool treats your resource as a black box. A human can use their knowledge to treat it at least partially as a white box. – CJ Dennis Mar 01 '20 at 22:16
  • Automated tools can be a good start, but it should never end with that. – Mast Mar 02 '20 at 06:43
  • How would such tools exist if nobody learned that stuff? – 12431234123412341234123 Mar 02 '20 at 20:27

5 Answers5

61

You have several assumptions here:

  • scanners can find all vulnerabilities
  • if a scanner cannot find a vulnerability then there are no vulnerabilities
  • all manual tasks can be automated
  • attackers would only use automated tools and not manual approaches
  • manual approaches cannot be turned into bespoke automated tools
  • finding vulnerabilities is the same as exploiting the vulnerabilities

None of these assumptions are universally true.

Automated scanners help make the process of finding vulnerabilities more efficient, but they are far from perfect and far from complete. Scanners are also not designed to exploit the vulnerabilities in a useful way.

In practice, you want to manually test the results of a scanner (false positives) and perform manual tests to look for things that the automated tool might have missed.

Attackers will use a mix of approaches and then often create or modify a tool to exploit the vulnerability so that it is repeatable and reliable. But that doesn't mean that the tool will work in other situations.

Automated tests are the basic threshold. If your site/program fails an automated test, then you've made a pretty bone-headed error and it should be fixed immediately (because it will be easy to find). But I've seen some cases where a developer has added a check for 1=1 in their SQL in order to hide from automatic scanners, but I was able to exploit the site using 2=2 (modern SQL scanners account for this now). I only knew that from manual testing and personal experience. You can't encode experience and intuition in a tool.

Coding is an insanely complex activity. That means that the errors can be complex, too. No tool could be created to look for or to exploit all vulnerabilities.

schroeder
  • 123,438
  • 55
  • 284
  • 319
  • 3
    Hahaha 2=2. This is a nice one. I understand experience is not to be neglected. Tell me, do you know if certifications like CEH have the right approach ? I'm really close to register for this cert lol don't wanna go there if there is another with a better more realistic approach – botanga Feb 29 '20 at 15:02
  • 3
    It's not that linear. CEH is a good cert if it meets your needs right now. It's not the best cert to use as your sole learning. There are many other certs, but they tend to be for people with more experience. CEH is one of those things where if you know nothing, it's fine, but once you learn a little more, you start to see why CEH has problems. Go in knowing that you will likely be replacing everything you've learned. – schroeder Feb 29 '20 at 15:18
  • 2
    Btw. the 2=2 example is also a nice lesson in another way. Not only do manual penetration tests help you in your capacity as security expert / penetration tester etc. to find exploits - if you only do automatic testing you also train your devs to beat the automatic tests instead of thinking about security in general, i.e. they produce dummy security exactly like that 1=1 thing. Regular manual tests that will involve some variation and if just "by accident" can counter that. – Frank Hopkins Mar 01 '20 at 05:20
  • 2
    Can you please explain what the `1=1` thing is about? Or tell me what to search to learn more? – Tomáš Zato - Reinstate Monica Mar 02 '20 at 09:59
  • 5
    @TomášZato-ReinstateMonica it is a typical SQL injection test: some queries are written to be open to new logic entered into the query, like `SELECT username WHERE userid = %variable%` which can become `SELECT username WHERE userid = "" OR 1=1` which would return all records. The `OR 1=1` was such a popular example of SQLi that scanners would use it literally and devs would block that specific string. Modern scanners now use random strings instead of hardcoded ones for testing. – schroeder Mar 02 '20 at 10:10
  • So just to clarify. The developer you mentioned deliberately added that to hide the vulnerability instead of fixing it? That's bad. – Tomáš Zato - Reinstate Monica Mar 02 '20 at 10:31
  • 3
    @TomášZato-ReinstateMonica that's right. The idea was "well, that's how you test and implement SQLi, so we'll stop that" – schroeder Mar 02 '20 at 11:48
  • Your first two bullets are logically equivalent FYI – Cruncher Mar 02 '20 at 16:34
  • 1
    @Cruncher you are absolutely correct. And I know they are. So, the question remains: why did I make them 2 different points ... ? – schroeder Mar 02 '20 at 16:43
17

DAST tools have several limitations that need to be addressed through manual inspection of the application:

  • They cannot find business logic flaws, because they do not understand the use cases and misuse cases of the application.
  • They only find known vulnerability types and patterns. Your mileage will vary based on the maturity level of the scanner, but even the best scanners won't find all the issues.
  • Complex patterns often have subtle security issues, like a custom implementation of an authorization protocol like OAuth. Only by studying the underlying architecture in detail, a pentester can find those flaws, while an automated tool will fail, as this type of analysis requires higher order reasoning.
  • Payloads to verify the exploitability of an application often have to be crafted for the specific application. Even if the scanner suspects a vulnerability, verifying it often has to be done manually.

That being said, a security scanner is a handy tool in the arsenal of a penetration tester, to identify low hanging fruits. The majority of the work will still be manual.

In addition, as a security tester you should always do whitebox testing, if possible. This includes having access to the source code, which gives you the possibility to use SAST tools as well. Those tools suffer from similar limitations, but may be able to find different types of vulnerabilities, and thereby add additional value. But even with DAST and SAST tools combined, you won't reach the required testing depth without manual testing, especially for applications with a high protection profile.

Demento
  • 7,249
  • 5
  • 36
  • 45
10

Here is a very recent topic that touches on the same issue: Why OpenVAS does not find all open ports compared to Nmap?. Takeaway: each tool is different, and may yield different results. Not to mention false positives and different testing methodologies.

Simply put, automated tools make educated guesses and interpret results. They get it right, most of the time. But you must understand how the tool works, what it does (and does not) and be able to tune it for optimal results.

A simple example: by default nmap, Openvas etc do not scan all tcp/udp ports but a selection of the most popular ports, that is a few thousands out of 65535. If you are not aware and run the tools with default settings, you can very easily miss active ports. For example, many sysadmins choose to run SSH on some random port rather than the standard 22.

The automated tools usually have plenty of options, and not just one button - so you have to understand what they do or you are shooting in the dark. Then, your audit is not in-depth and has little value, because you don't know what you are doing, and what you should be looking for. All you've done is scratch the surface and look for the most obvious flaws.

Put it differently, why should we hire professional pentesters if all it takes is downloading and running a tool ? Because a competent pentester has experience and will go farther and can find vulnerabilities than a script kiddie will miss.

It is rarely "as simple as running a tool".

A properly-configured machine that is exposed on the Internet should have some kind of defense mechanism built-in: a firewall and/or IDS that will thwart this kind of reconnaissance effort.

When they detect port scanning activity, they typically react by blocking your IP address, or they throttle the traffic, drop some packets selectively or choose to return deliberately misleading results to frustrate hackers. You end up with incomplete or downright bogus results.

Keep in mind that tools like nmap, Acunetix etc are noisy and usually very easy to spot (and block) by an IDS because the traffic they generate has typical signatures and patterns. So unless you are testing a machine that is unprotected or loosely protected (on a LAN perhaps), you'll have to tune them quite a bit to achieve meaningful results.

So the answer is that you do both: you use automated tools and then you do manual testing, especially when the tool detected something, like an open port but was not able to exploit it, or you want to double-check.

Kate
  • 6,967
  • 20
  • 23
  • Over the course of a year or even a few days you can probe every port... and never once be detected. Shifting your IP, MAC, probe port, time delays- it is cat and mouse, and if you (work) for the long game then what's a few weeks anyway? – J.Hirsch Mar 02 '20 at 18:12
6

Vulnerability scanners can't find every vulnerability that may be present. They basically look for patterns and exploitability of already-known vulnerabilities.

Furthermore, you may get false positives or other unexpected output when you use automated tools. So you need professionals who can evaluate the output and conduct more advanced pentesting.

secprof
  • 61
  • 1
  • What is "more advanced pentesting" ? Is CEH advanced pentesting ? Is any certification really advanced ? Of more advanced means highly skilled reaserchers ? I'm asking because almost all certifications give the same material, or the results of researches that are packed into tools. So in the end, it seems to always come down to find the right tool – botanga Feb 29 '20 at 13:18
  • 5
    @botanga CEH is an entry-level certification - and a really bad one at that. –  Feb 29 '20 at 13:24
  • 2
    @botanga no, it does not come down to the right tool. It comes down to the right approach. Sometimes the right approach involves a tool. – schroeder Feb 29 '20 at 14:42
  • OMG. I was thinking on investing a gran on CEH. Do not tell me I'll regret it lol – botanga Feb 29 '20 at 14:53
  • @botanga In my experience no certificate, diploma or degree can replace someone with real talent in finding ways to exploit the system. To develop talent in this is the same as developing talent in anything else: skateboarding, programming, painting etc - you need lots of practice and you need a passion for it (because doing something that don't interest you repeatedly gets boring quickly). To some going to school (University, art school, law school etc.) is one way to get some practice and get people to guide you. For some taking lots of tests is one way to prove they've got the skills – slebetman Mar 01 '20 at 04:00
  • ... A certificate or diploma or degree is just one indication to others (and sometimes yourself) that you know what you're doing. It does not mean you're good at it – slebetman Mar 01 '20 at 04:01
0

Here're some common difficult challenges (especially from the reconnaissance and pivoting activities) to application security automatic scanners that they usually fail to detect or they could have false positive detections:

  • Hidden paramater detection for further scanning
  • Hidden or relatively referenced URL detection for further scanning
  • Business/logic security impact to identify a vulnerability from potential false positive detections. Threat with CSRF attacks belongs to this kind of challenge. Scanners also have problems with the vulnerabilities only shown when a specific value of a parameter is set.
Minh-Triet Pham Tran
  • 500
  • 3
  • 7