9

Fuzzing is a convenient, relatively low-cost way to detect some kinds of vulnerabilities, particularly in C/C++ code.

My question: How much fuzzing is enough? Are there are any standards or best practices?

Example: Microsoft's SDL requires that applying fuzzing to security-sensitive code that parses file-based input. The SDL requires a minimum of 500,000 iterations of fuzz testing, and at least 250,000 "clean" iterations. (Here's a citation. Any fuzz iteration that finds a security bug resets the count of "clean" iterations back to zero.) The SDL also requires that all network data be fuzzed for at least 100,000 iterations. (Here's a citation.)

Does anyone know of other guidelines, or the criteria that other organizations use?

AviD
  • 72,138
  • 22
  • 136
  • 218
D.W.
  • 98,420
  • 30
  • 267
  • 572
  • 3
    here's an article from Google Security that may be interesting, if not answering your question:- http://googleonlinesecurity.blogspot.com/2011/08/fuzzing-at-scale.html – DanBeale Aug 13 '11 at 13:22

1 Answers1

7

I don't have numbers for you, or specific guidelines, but I think an important issue to point out, but was not sufficiently emphasized in those links, is Code Coverage.

It makes little sense to run a huge number of fuzz iterations, if they are all similar - from a code path point of view.
That is, "dumb fuzzing" may generate many different inputs (file/network/RPC/whatever), that look sufficiently different, but all have the same effect on the code, causing the program to take the same code paths. Obviously, this is not very effective.

Microsoft has also developed internal tools (I know of FuzzGuru) that implement "smart fuzzing". (See this talk from a few years ago discussing it). This enables MS to run much more efficient fuzzing, with much lower numbers of iterations (unofficially, of course) - simply because they can ensure much higher percentage of code coverage at each iteration.

In any event, it is important to do your fuzzing - whether "smart" or "dumb" - in combination with code coverage tools, to ensure you're getting appropriate value out of your fuzzing efforts.

AviD
  • 72,138
  • 22
  • 136
  • 218
  • OK. Any guidelines or metrics on what level of code coverage one should try to achieve? How much coverage is enough? 50%? 60? 80? – D.W. Aug 10 '11 at 21:25
  • Umm... no, I actually dont, sorry (I should have expected this question). MS aims for >80% where possible, though I have seen numbers that suggest much lower than that (40-50%...). Fuzzing is supposed to improve that, though... Code coverage might be seen as more of a Testing discipline, though, we kinda "borrowed" it... – AviD Aug 10 '11 at 21:47