There is no one way to find vulnerabilities. But here are some steps you can follow.
Target
First you need to choose a platform and a piece of software to attack. To begin I would choose something that is open source. There are several advantages to this; the main one being that you can look at the source code. You then need to pick an aspect that you would like to attack. For example, maybe you want to attack the UDP implementation of the Linux networking stack.
Performing an analysis on a closed source piece of software means you're disassembling the binary, rooting through instructions, and debugging the process. This is long and tedious. Better to get a grasp as to what breaks code with source code before you go looking for it in disassembly.
By being specific in your target allows you to systematically analyze a piece of software.
Analyze
With your target in mind begin your analysis of the portion of the software you want to find vulnerabilities.
- Determine which source code files affect your target.
- With open source you can insert debug messages to ensure you understand the code flow. This can be extremely important. Knowing what sections of code are called, and the variables that lead to that outcome is key in understanding what is going on.
- Run code analysis tools over the project. Depending on the project this might be a moot point, but they can be handy and catch common programming errors.
- Enable all of the compiler build flags. Your goal is to find programming errors. What better way than to have the compiler tell you where it thinks the code is bad.
These are just a few of the things you can do to analyze the software. Build a list of possible coding errors.
Triggering
Now with a list of possible coding flaws you need to determine if you can trigger them. Again, debug messages will help you. Go back to the source code and determine what exactly needs to happen for each coding flaw to break the software. You're not looking for full exploitation, you just want the code to crash, or do something unexpected. You need to determine what could trigger a coding flaw. This could be anything from affecting a length variable, tricking a function to take a path to process data incorrectly, etc. Some coding flaws just aren't triggerable, but that's the nature of vulnerability analysis.
At this point you have a list of flaws, and a list of ideas for each flaw on what might trigger it to do something unexpected.
Fuzzing
Now you write code. Using pretty much whatever programming language is convenient for the software you're attacking. You could write Python code to throw specific packets at network devices to attempt to take down the UDP implementation of a Linux based device.
The goal is to implement your triggers, and hope that the code works the way you think. Your debug messages will be helpful here.
- They can tell you if the code path taken is abnormal.
- They can show you variables that you're attempting to manipulate
- They ensure that your trigger is doing what you expect, and you can adjust it accordingly.
With any luck you're able to cause something different to happen. Maybe that can lead to code execution, maybe not. That's a horse of a different color.
Reality
Vulnerability analysis takes time. A lot of time. You're not going to spend a day analyzing software and find 10 vulnerabilities. The unofficial average for vulnerability analysis is 1 vulnerability per 3 months of analysis. You can double that time if you're analyzing a non-open source project.