W3af and automated vulnerability scanners vs manual testing

Question

How effective are tools such as w3af in looking for web app vulnerabilities compared to looking for vulnerabilities manually? Are they able to find all vulnerabilities from OWASP top 10 such as reflected xss, persistent xss, sqli, lfi/rfi, and unrestricted file upload? Or will some vulnerabilities fall through the cracks and remain undiscovered?

Answer 1

Manual testing:

• Slow;
• Humans can forget some important tests;
• Complex, sophisticated and creative testing.

Automated testing:

• Fast;
• Tests a lot (thousands) of vulnerabilities (never forget a test);
• Easily repeatable after remediation;
• "Cake Recipe" (will just test what it was designed to test).

Based on the above points, we can say that each type of testing have its own positives and negatives sides.

Are they able to find all vulnerabilities from OWASP top 10 such as reflected xss, persistent xss, sqli, lfi/rfi, and unrestricted file upload?

A lot of Web Security Scanners were designed to find all the above types of vulnerabilities, but nothing can guarantee that they will find all instances of the above vulnerabilities.

Or will some vulnerabilities fall through the cracks and remain undiscovered?

Some vulnerabilities can fall through the cracks, no tool is perfect.

Answer 2

Tools are always limited in my opinion. Testing an authorization mechanism and session fixation (I've yet to come a cross a tool that can do this) are just two examples a tool can't (properly) do. Tools are good for the so called low hanging fruits (most of the OWASP top 10 issues), but can't be called a penetration test.

I generally use tools to get a global indication of the system. While running in the background I perform manual tests. In my experience as a security consultant, the number of vulnerabilities discovered by web application scanners is always less than what I find manually.

Another thing is combining vulnerabilities, for example:

1. Session cookie does not have the httpOnly attribute set
2. Application is vulnerable to CSRF attacks
3. Application is vulnerable to Persistent XSS attacks

While no 1 by itself can be considered low and no 2 and 3 are high, combining the three will lead to session hijacking, which can be considered critical. This is something a tool will not do.

So, yes most vulnerabilities will discover OWASP top 10 flaws. And yes, it is possible that some vulnerabilities will not be discovered, this can be due to time restrains.

Also keep in mind that what's considered sufficiently secured today can be non-secure next month.

Answer 3

How effective are tools such as w3af in looking for web app vulnerabilities compared to looking for vulnerabilities manually?

They are effective for what they are. Scanners will find the low hanging fruit.

However, they typically fail at finding vulnerabilities caused by flawed business logic. In this area humans have the advantage.

Scanners typically test 10's of thousand of vulnerabilities which is not feasible for a single person to accomplish. In this area scanners have an advantage.

Are they able to find all vulnerabilities from OWASP top 10 such as reflected xss, persistent xss, sqli, lfi/rfi, and unrestricted file upload?

Yes.

Webinspect has an OWASP scan if you want to restrict scanning to the OWASP top 10.

Or will some vulnerabilities fall through the cracks and remain undiscovered?

Yes.

Scanners will not always find the same amount or type of vulnerabilities every time. At least this has been my experience using Webinspect for over 10 years. Meaning that scan result "A" may contain 10 instances of XSS where scan results "B" may contain 7 instances of XSS.

Usually when I see this happen it's the same page in case A and B, but B lists fewer affected parameters.