What you are doing is called reverse engineering an executable. It is not unusual for a person who is hard coding a password into source code (a horrible security strategy) to make an amateur attempt to keep it from appearing in the static data intact. However, the password PAssw0rd may not be at all related to the actual one. The various program responses are obviously intentional, so PAssw0rd may be a deliberate red herring.
Several tools are common in reverse engineering efforts.
- The command "file" which takes the file path as the first parameter so you can determine (in most cases) what type of executable you have.
- Disassemblers which show EXACTLY what the executable does but is difficult to read for those that don't write assembly code on that specific architecture or have experience with disassembly.
- Decompilers like Boomerang, Hex-rays, and Snowman can provide some greater readability but they do not recover the actual variable names or syntax of the original program and they are not 100% reliable, especially in cases where the engineers that created the executable tested with these packages and tried to obfuscate the security further.
- Data flow diagrams or tables. I know of no free tool to do this automatically, but a Python or Bash script over the top of a text parser of the assembly output (which can be written in sed or Perl) can be helpful.
- Pencil and paper, believe it or not, for jotting flows and ideas
A common approach is to hunt for two things in the decompiled or disassembled code.
- Cryptographic functions, such as a hash or cipher algorithm
- The user prompt for the password and the subsequent read of the
user entry
These are the two endpoints in the data flow. You will want to back-trace from the place where the password enters the cryptography algorithms and forward-trace the user entry of the password, hoping to find where the two ends meet.
Once you trace the path between user entry and authentication components, you will be able to see the mechanism in between. If the security was done correctly, it will be a properly created hash value and your attempt will end poorly, but I suspect, this being an exercise, favorable results will be possible.