My questions is related with static code analysis approach used by Veracode vs Fortify/AppScan.
A) Is there an advantage/disadvantage of conducting reviews of binaries over source-code? B) Which one provides more coverage/vulnerabilities. C) Any examples would be great.
Thanks,
A) Is there an advantage/disadvantage of conducting reviews of binaries over source-code?
Compilers often DO NOT write code expressly as intended in the source. For example, Return Oriented Programming exploits the fact that compilers will insert many more RET opcodes than the programmer is aware of. Due to pipelining and other optimization tricks, compilers essentially rewrite your code and can possibly add vulnerabilities of their own. This means that some constructs in the source code may not be expressed in the binary at all!
This is a class of error that is essentially impossible to catch through manual code analysis... and I suspect the risk would still be there for Java/C# JIT code as well, but this would be obfuscated from a static analysis tool.
Manual source code analysis can help by the fact that most common vulnerabilities can be caught by visual inspection. And it has other benefits as well, most notably in just decreasing the number of prod bugs in general. (And thus cost.) And it helps the social aspect as well: source reviews encourage people to write as if someone else is watching. A major disadvantage is that if you're dealing with a dynamically typed language, such as PERL, Groovy, LISP and its derivatives... most data won't be looked at until runtime which means neither static analysis or source code analysis is sufficient.
You can even fool static analysis tools by converting statically typed code into instructions at runtime. Veracode doesn't like your java construct? Rewrite it using reflection, and the Veracode bug disappears and no one is the wiser. As a security expert you also need to assume that the programmers working in your own company are a potential threat.
So in short, there's no way to escape Source code analysis, static analysis, and dynamic analysis in any given application if your hope is an acceptably secure code base.*
B) Which one provides more coverage/vulnerabilities.
Source-code analysis depends on the reviewers having excellent security skills and since they're human beings, you can't reasonably expect perfect performance. That's why static analysis tools come in handy. My preference is to run static analysis tools first and then do the secure code reviews after the discoveries are mitigated, I prefer using STRIDE but there are other frameworks to consider. This lets humans concentrate on the less mundane security problems that are more logic-related. But a smart human will beat a dumb static analysis tool any day of the week.
Source-code analysis can help you find where a programmer left behind a back door... static analysis tools aren't equipped to handle that.
So the answer to this question is... "it depends." What kinds of threats do you want to go unchecked?
*Unless "acceptably secure" is defined as leaving the windows open and the front door unlocked.
One could write books about these things and not ever be satisfied. Let me try to answer this in general, without naming special products.
This is mostly orthogonal to the way the tool operates and is largely a measure of the quality of that product. With enough effort, all tools could implement all checks for all kinds of vulnerabilities, but no one has enough manpower lying around. Things that are easier to do will be done; those that are hard to do might never be implemented.
You could have two source code analysis tools that focus on totally different things (e.g. one on race conditions, the other on overflows). I am sure out there between the tools there is always some overlap, but no two tools find exactly the same kind of flaws.
If you have the possibility, run as many of those as you can, you never know if you have a bug somewhere that can only be found by one of the tools you did not run yet...
The short answer is use them both. Hybrid or Integrated Application Security Testing (IAST) combines Dynamic Application Security Testing (DAST) of binaries and byte-code with Static Application Security Testing (SAST) to create a unified view of an application’s vulnerabilities. The goal is not just to find vulnerabilities but also help developers quickly identify, interpret, and fix those errors. Hybrid tools go beyond DAST using static analysis to point to the specific lines of source code and to extend the tool’s coverage. Moreover IAST tools reduce false positives – the bane of software developers - with live tests on SAST findings. IAST can take advantage of strengths of both SAST and DAST methods and offset the weaknesses inherent in each. For example, dynamic testing of an SQL injection can confirm that the attack indeed made it to the database. IAST speeds up testing by examine the code to narrow the attack surface; there is no need to test a page for SQL injection if the page doesn't do any SQL. IAST can employ SAST to discover attack surfaces in code that may be hidden to dynamic testing methods.
Examples of IAST tools are Contrast from Aspect, Quotium Technologies’ Seeker, Appscan Enterprise from IBM, Acunetix Web Vulnerability Scanner and HP WebInspect Real-Time. See Jeff Williams' blog post Why static application security scanners just cant cut it anymore See the similar question Effectiveness of Interactive Application Security Testing
A) Is there an advantage/disadvantage of conducting reviews of binaries over source-code?
I normally "decompile/disassemble" the binaries and review them instead of the source code. Sometimes I don't even have access to the source code. There could be different "branches" and even source that never makes it in the compiled binary (dead code).
B) Which one provides more coverage/vulnerabilities. Source code has hints, like //TODO and comments that can help if something needed to be implemented/have weaknesses. Binaries do not. Both have there merits.
