Best way to triage crashes found via fuzzing, on Linux?

Question

When doing fuzz testing, it is easy to end up with many bugs (many crashes). This makes it important to have a way to triage each bug that's detected, so we can prioritize them and focus our effort on the ones that are most likely to represent exploitable security vulnerabilities.

Are there any tools to automate the triaging process? I'm most interested on tools for Linux.

On Windows, we have the !exploitable analyzer, which analyzes a crashing program execution to determine whether it represents a bug that is probably exploitable, probably not exploitable, etc. Is there anything like this for Linux?

Answer 1

Here's the best heuristic I know of. Run the program under Valgrind memcheck, and then look at the warnings that Valgrind outputs. We can classify them into a couple of categories:

• Invalid write: Look at the address. If the address is small (say, between 0x0 to 0xFFF), then this is a NULL pointer dereference: probably not exploitable, low priority. Otherwise, this is a write out-of-bounds: potentially exploitable, a serious high-priority bug.

• Invalid read: Look at the address. If the address is small (say, between 0x0 to 0xFFF, say), then this is a NULL pointer dereference: probably not exploitable, low priority. Otherwise, this is a read out-of-bounds: could be exploitable if you're unlucky, but often these bugs are not exploitable; call it medium-priority.

• Invalid free(): This could be a double-free bug, and there's a significant chance it could be exploitable. High priority.

• Mismatched free() / delete / delete[]: Potentially exploitable, depending upon the circumstances. Medium priority.

• Conditional jump or move depends on uninitialised value(s): While such bugs can sometimes be exploited, exploitation is by no means guaranteed, and it is not likely to be easy. Often, these are benign false positives. Medium to low priority.

• Syscall param ... points to uninitialised byte(s) or Syscall param ... contains uninitialised byte(s): Same as above. Medium to low priority.

• Source and destination overlap in ...: Very unlikely to be exploitable. Low priority.

• Memory leaks: (e.g., Still reachable, Definitely Lost, Indirectly Lost, Possibly Lost): Very unlikely to be exploitable. Low priority.

This is the best heuristic that I know of. I don't know of any tool that implements it for you, but it's not too hard to script up yourself. Does anyone know of a better triaging heuristic or tool for Linux/Unix systems?

Answer 2

I like using the peach fuzzing platform. This contains a testing harness which will record memorydumps from crashes and link them to the fuzz test case. When the process crashes, the testing harness will restart it and continue until testing is complete.

As far as I know !exploitable is pretty unique. Valgrind is useful at determining flaws like dangling pointers. The old fashioned way of determining if a crash is exploitable is by looking at the EIP, 0x41414141 is always a welcome sight. However its possible that the application will crash before the function returns because you have overwritten a pointer on the stack. So even a crash on read/write to general purpose register like ebx 0x41414141 maybe potentially exploitable and the process is just crashing prior to its return. Make sure you look at the callstack to see if it has been corrupted.

Answer 3

CERT's open source Linux Triage Tools can be used for triage of bugs found via fuzzing. The tools include a GDB extension that is similar to MSEC's !exploitable, but for Linux.

http://www.cert.org/blogs/certcc/2012/04/cert_triage_tools_10.html

Answer 4

Some of American Fuzzy Lop's sister projects have some tools that might be useful. In particular:

• afl-crash-analyzer has some automation to help analyze a crashing test case, including running the exploitable GDB script to test whether the crash seems exploitable.

• crashwalk has some automation to test which crashes are reproducible and to triage them using the exploitable GDB script for Linux; it also has a similar script for Mac OS X.