4

I have a program that reads data from a file, parses/organizes it, and spits it out as an encrypted XML. The application can also take an encrypted XML and spit out the original file. My objective at this point would be to access the clear-text XML (I'm not interested in the clear text original file as it's not XML organized)

I have no idea what the encryption is yet although one guy on a forum said it was AES-128 (not sure how he got to that conclusion).

I ran PEiD with the KANAL plugin on the application, it doesn't detect any encryption signatures.

Because I have access to the program and some past some experience with exploiting BO on WinXP with some knowledge of ASM, I figured I could give it a try using a debugger.

In a nutshell, what are the general steps I should be following to figure this out? In this situation would it be best to start looking for the encryption key itself, or find a way to use the application's encrypt/decrypt functions to my advantage?

EDIT: Target is a Windows executable

Juicy
  • 1,407
  • 4
  • 16
  • 31
  • 2
    @Juicy Not to imply that this is off topic, but you may get a better response at reverseengineering.stackexchange.com – amccormack Jun 17 '15 at 22:01

1 Answers1

5

Reverse Engineering a Windows Binary

Get a Disassembler

I'd start by downloading the evaluation version of IDA. Because it is the evaluation version it has a pop up and won't let you save, but it should be good enough to get started. Note: This won't work for a 64-bit binary. Go check out this page for alternative disassemblers if necessary.

Inspect the Strings

Before you dig into the actual code, start by inspecting the strings. IDA has a tab (or an option in the "views" menu) that will list all the strings found in the binary. Oftentimes, encryption libraries will accept a string like "SHA-256" to indicate what kind of algorithm to use.

Additionally, look for anything that could be a key. The kind of key being used depends on the algorithm, so you may be looking for a password, passphrase or not a string at all but a block of data.

If you can find the name of the encryption routine as a string, you can cross reference it (the x key in IDA) back to the original encryption function. If you can find the encrytion function make note of the address.

Dynamic Analysis

Some of the Disassemblers mentioned above (including IDA) also include a debugger. I personally like OllyDbg. Set a breakpoint at the encryption routine you fund above and then run the program. Inspect memory and attempt to find the encryption key.

You can also use your debugger to dump the memory of the program. After dumping the memory of a live program, you can use a utility like strings in sysinternals to attempt to locate a key.

Community
  • 1
amccormack
  • 3,971
  • 1
  • 15
  • 23
  • Thanks, your answer helped me a lot for initial investigation. I ended up running the program through PEiD which told me the app was .NET. I then ran it through GrayWolf which showed me the whole C# code alongside the opcodes. Was then just a matter of following the trail until the encryption function and tweaking the opcodes. – Juicy Jun 18 '15 at 20:11
  • Ah, I would have given an entirely different answer if I knew it was a .Net application. I've enjoyed using JetBrain's [dotPeek](https://www.jetbrains.com/decompiler/) for .Net reversing. Congrats on solving though. – amccormack Jun 18 '15 at 20:13