14

Is running an automated application vulnerability scanner sufficient to certify our application is secure? Why isn't it?

From the original Area51 proposal.

AviD
  • 72,138
  • 22
  • 136
  • 218
  • Also see http://security.stackexchange.com/questions/215/automated-tools-vs-manual-reviews and http://security.stackexchange.com/questions/214/white-box-vs-black-box that discuss different aspects of this issue (but not quite duplicate). – AviD Nov 16 '10 at 06:19

4 Answers4

7

Sure automated checks are not sufficient, and believed that never will be. Well, until some AI is made. As for example fuzzers - they can cover only checks that fails for harsh input. Logical bugs will be missed. Source code analysis shows good results when the talk is about really severe bugs and those vulnerabilities that requires several conditions to meet. Binary analysis might be daunting for not so experienced person, but it is possible to inspect closed source code application and besides, it takes into account compiler specifications that are impossible to diagnose during source code analysis. So, every method has it advantage and should not be used alone.

  • 5
    I agree with this answer. In addition - using automated tools, you won't be able to check the security flaws in application business logic. Combination of automated and manual testing is IMO the correct way. For example in OWASP Application Security Verification Standard, automated testing is just the first level of cheking if application is secure: http://code.google.com/p/owasp-asvs/wiki/Approach – bretik Nov 12 '10 at 14:12
3

Relying only on automated vulnerability scanning to vet the security of most applications is woefully inadequate.

Why? Because automated tools will miss vulnerabilities (type II errors/false negatives).

The uglier implication of this question is the application’s security is important yet it was ignored during development.

Tate Hansen
  • 13,714
  • 3
  • 40
  • 83
1

Applications cannot be certified as secure in the same way that vehicles cannot be certified as safe to drive. Your mechanic may hand you a list of things that he or she has checked and perhaps even verified as safe. However, that does not mean that something will not go wrong once you get your car on the road or track.

Only mechanics that seek to rip you off will provide a fully automated safety inspection. It would be insane to think that in the next 50 years it would be ok to consider any application fully secure or safe, even with artificial intelligence. Even formal methods with formal specifications have been known to fail.

I suggest that you properly scope an application's unique situation before you determine the outcomes.

atdre
  • 18,885
  • 6
  • 58
  • 107
  • 1
    Which I think in effect answers the question perfectly in that what you are saying is that it is impossible to rely on only automated vulnerbility testing software. – Anonymous Type Nov 16 '10 at 23:23
  • I, personally, feel that automated vulnerability scanners, whether the network, application, or code kind -- all miss like 99.9% of vulnerabilities and report information about false vulnerabilities (or false information on them) 99.9% of the time. The only data I have to back this up is the NIST SATE data from 2008 and 2009 – atdre Nov 17 '10 at 21:52
1

Sufficient for what?

Sufficient to defend against other purely automated attacks like worms? Probably.

Sufficient to defend against targeted attacks by skilled attackers? Certainly not.

A better question is "Does an automated scan mitigate risk to an acceptable level for my application?" The answer is, of course, it depends...

To use an analogy, you wouldn't ask the question, "Is going to my General Practitioner sufficient?" without the context of "I need a yearly checkup" or "I just chainsawed off half my torso."

An automated scan might be sufficient for a low criticality brochureware site on its own domain, but is definitely not sufficient for a banking app. Most sites fall somewhere in the middle, so... It depends ;-)

spinkham
  • 422
  • 2
  • 3