Fuzzing and its impact on testing environment

Question

I asked this question on StackOverflow but got no answers so I thought that I would try my luck here since fuzzing is closely related to security and often used in vulnerability assessment tests.

I'm currently writing a fuzzer that will generate a payload based on a custom specification format.

All is fine and I'm happy with the first version I wrote but I encountered an issue when trying to apply this fuzzer to a task that wasn't my initial use case.

The issue is related to how the fuzzer's input will affect the testing environment.

E.g:

• If the tested method constructs a buffer with variable size what would happen if the generated value is 2^64 and would trigger an out-of-memory (OOM) error?

• If the test method removes a file specified by a variable filename what would happen if the generated value is * and the whole directory would be deleted?

Of course these are just some naive examples but the point is that I'm trying to find a way to reduce the consequences of fuzzing on the test environment (while still remaining portable if possible).

This would make the fuzzer safer to use but also more fluent because fuzz test cases could be easily chained and run in parallel without breaking or corrupting the environment every other payload.

There are some radical solutions like:

• not using fuzzing on methods that could have critical side effects and consequences
• black-listing / white-listing certain inputs in certain use cases

Both would be really limiting in my opinion and make it difficult to use the fuzzer generically on black-box fuzz testing.

Other possible solutions like:

• running tests in a headless VM like Vagrant
• running tests in a Linux container like Docker
• intercepting certain syscalls like I/O calls and sanitize their input

are valid but all suffer of the issue that they are very high-level. Syscalls and Docker also have the problem of not being very portable.

How do more professional / enterprise fuzzers handle this issue ?

I would imagine that black-box testing software probably solved this issue in a portable way, didn't it?

Answer 1

Any testing, not just fuzzing, should be performed in some sort of sandbox (eg. a virtual machine). This has two main benefits: 1) you can quickly reset things to a known configuration, and 2) bugs (such as a runaway recursive deletion that includes .. in the list of directories) can't harm important data.

Answer 2

The Sulley Fuzzing Framework was announced along with the book "Fuzzing: Brute Force Vulnerability Discovery" and more information about both are available on fuzzing.org

Other fuzz-testing frameworks, probably before Sulley, and a few after, utilize a guest VM technique. There are also VMs to test the hypervisors themselves such as Tavis Ormandy's iofuzz ISO. Another technique mentioned in the book is keeping track of faults, crashes, and state (such as packet order and sequence), so it's worth a full read.

Potentially the most-current book on fuzz testing that covers Sulley is the latest fourth edition of "Gray Hat Hacking: The Ethical Hacker's Handbook". If Sulley or the resources from these books isn't helpful, perhaps you can help me focus in on exactly what you are looking for.

While there are Enterprise fuzz testing suites, engines, and even appliances -- I might suggest instead consulting with the people who write and run specialized fuzz tests like the one you are interested in and looking to build. For example, Deja Vu Security has scaled virtual machine fuzz testing engines for many years. The veteran expertise from their researchers can likely be found on their blog, in their codebases, and throughout their talks at DEF CON, BlackHat, and other information security conferences.