7

Black-box penetration testing is one to check a web application for vulnerabilities. It can find some vulnerabilities, but not all.

What fraction of vulnerabilities does black-box pentesting find, on average? Is anyone aware of any data or evidence on this?

AviD
  • 72,138
  • 22
  • 136
  • 218
D.W.
  • 98,420
  • 30
  • 267
  • 572

2 Answers2

10

One paper in the literature provides some partial data on this subject [1].

The paper analyzed 9 web applications using two methods: (1) black-box penetration testing, and (2) manual code review by an expert. It compared how many vulnerabilities were found by each.

In total, the analysis found a total of 91 vulnerabilities. 39 of them (43%) were found by black-box penetration testing. 71 of them (78%) were found by manual code review. There's no way to know how many more vulnerabilities might be present that neither technique discovered.

Therefore, one possible conclusion is that black-box penetration testing finds less than half of all vulnerabilities present in a web application, on average.

Another conclusion is that the combination of both methods seems to be better than either one alone; they each detect some vulnerabilities that are missed by the other. In the paper mentioned above, neither method alone was enough to detect all vulnerabilities: manual review found 52 vulnerabilities (57%) that were not detected by black-box pentesting, and black-box pentesting found 20 vulnerabilities (22%) that were not detected by manual code review.

However, this paper has a number of limitations. It looks at only 9 web applications. It is hard to know whether these are representative of all web applications in the wild. Manual code review was performed by only one reviewer. Black-box pentesting was performed by only one tester, and using only one tool. It is not clear whether the effectiveness of these methods might vary from person to person, or from pentesting tool to another. Therefore, this data is only a first glimpse, and can't be treated as the final answer.

[1] Matthew Finifter, David Wagner. Exploring the Relationship Between Web Application Development Tools and Security. WebApps 2011.

D.W.
  • 98,420
  • 30
  • 267
  • 572
  • 2
    These numbers do compare roughly with my experience. As you say, there are also additional bugs that are not found by either (but e.g. may be found by design review, or threat modeling. Or not.) Another important point to note is methodology and tools. For example, simply running a web vulnerability scanner is likely to found approx. 45% of the vulns you could find in a manual pentest (given enough time). Static analysis (source code analyzers) have even worse statistics. – AviD Mar 14 '12 at 16:59
0

This question would be difficult to solve, barring the Halting Problem.

I would suppose you could say that a smaller or larger percent compared to other techniques, but usually never the same, zero, or all (especially not with the same set of eyes during the same sort of time frame).

Update for clarity:

A better question would have been, "What percentage MORE vulnerabilities are found when app pen-testing is combined with source-code assistance and commercial SAST/IAST than just plain DAST black-box?"

atdre
  • 18,885
  • 6
  • 58
  • 107
  • Thanks for the feedback! Please don't hesitate to provide any statistics you've got that address your preferred version of the question, if you have 'em. I'd warmly welcome any data or evidence you've got. – D.W. Mar 15 '12 at 05:05