Context:
I want to reverse engineer a protocol in place for the Nintendo 3DS. The implementation however, uses TLS, which makes it difficult to reverse engineer. I'm currently able to sniff packets to/from my 3DS because I have a spare router being used for a "testing wifi" that runs through my desktop's ethernet, and then through my desktops wireless card to the actual internet.
What I've investigated:
- MITM
- Proxies
Roadblocks:
- I can't force the 3DS to accept a questionable certificate.
- I can't forge the certificate.
- I won't attempt thieving the actual private key ( I have little interest for it once I know the protocol ).
- I'm unable to read the session key from my console's RAM, due to lack of hardware.
Potential solution:
- I'm almost 100% positive that the protocol is some form of HTTP ( Probably through headers, mostly ). So if the encryption is XOR based, I may be able to get part of the key, at least? I don't know a lot about cryptography.
- I've thought about setting up a proxy, a CA server instance [ and pointing my desktop's host file to this server instance ], installing Nintendo's certificate and then overwriting their public key with my own private key, and then using my 3DS's certificate on the proxy to communicate with Nintendo. I'm not for certain that this will work as I honestly don't have a complete grasp on how TSL works beyond that it's not something you can break easily.
- Is brute-strength a viable option if I am able to generate more encrypted data?
tl;dr- Is there a way for me to decrypt this data without the server's private key, or cracking into the RAM to find the session key? I do have the server's certificate as well as my 3DS's client certificate.