Decrypt SSL traffic with the openssl command line tool

Question

I'm trying to do some manual data extraction/encryption/decryption with the openssl command line tool.

I've been reviewing RFC5246 to work out what I need to do. It is not crystal clear to me if I am going to be able to do this step with that tool. I'm assuming the private key in the explanation is the private key generated when I created my self signed certificate.

When RSA is used for server authentication and key exchange, a 48- byte pre_master_secret is generated by the client, encrypted under the server's public key, and sent to the server. The server uses its private key to decrypt the pre_master_secret. Both parties then convert the pre_master_secret into the master_secret, as specified above.

Can someone tell me if my assumptions are correct. Can the openssl command line tool be used and supplied with my server private key and encrypted pre_master_secret from the client to generate the pre_master key for the server so it can be used to create the master key?

If so, I am not sure how to do it as I'm not very familiar with the tool.

The other thing I should point out is that the cipher suite I am working with is TLS_RSA_WITH_AES_256_CBC_SHA and I can see in Wireshark that the pre_master_secret from the client is 256 bytes long.

Answer 1

I'm not entirely sure but I think the answer is no. The openssl command line client is a heterogeneous collection of tools. The X.509 commands can be useful to manipulate certificates, but the cryptography commands are rarely useful for anything other than testing OpenSSL itself.

If you need to do cryptographic calculations with common algorithms, I recommend the Python interactive command line with the Cryptodome library.

But to decrypt SSL connections, the easiest way is usually to use Wireshark. Tell Wireshark where to find the private key and it will decrypt a TLS connection that uses RSA encryption. For connections using ephemeral Diffie-Hellman, you can't decrypt the traffic with the key alone, you need additional information from either the client or the server.

---

Note that using the TLS_RSA_WITH_AES_256_CBC_SHA ciphersuite is a bad idea for several reasons:

• It doesn't have forward secrecy, so if the server's private key is ever compromised, all connections made with this key are also compromised. Ciphersuites that use a Diffie-Hellman key exchange (with EDH or ECDHE in their name) have forward secrecy.
• It uses RSA decryption, which involves padding, which is a classic source of implementation bugs and leakage through side channels. Ciphersuites with EDH or ECDHE in their name in addition to RSA, or with DSA or ECDSA, use signatures instead of decryption and are less likely to suffer from implementation defects.
• It uses CBC decryption, which involves padding, which is a classic source of implementation bugs and leakage through side channels. Ciphersuites without CBC in their name are less likely to suffer from implementation defects.

Answer 2

RSAES-PKCS1v1_5, which SSL/TLS-through-1.2 plain-RSA keyexchange uses, can be decrypted by OpenSSL commandline operations rsautl or pkeyutl (the latter since 1.1.0 in 2010). See their respective man pages, which should be available on your system if not Windows, or online.

Note this gives you the premaster secret, which by itself cannot decrypt (or authenticate) traffic. You must use the premaster plus nonces to derive the master secret, and then the master secret plus nonces to derive the working keys (plural). The derivation function 'PRF' differs between SSLv3 (no longer used), TLS 1.0 and 1.1 (RFCs 2246 and 4346), and TLS 1.2 (RFC 5246). (PRF will differ again in TLS 1.3 as drafted, but that will also eliminate plain-RSA keyexchange entirely.)

Commandline cannot access the SSL/TLS PRFs directly, but can do the HMACs from which they are built (except SSLv3); see the man page for dgst in the same places as above, and note -hmac $key can only handle byte sequences that can be passed from shell/etc, which is not all, so you may need -mac hmac -macopt hexkey:$hexkey instead.

That said, I concur with Gilles it's much easier to let wireshark do it. If your problem is you have the data in some form other than a capture file, wireshark distros come with several auxiliary commandline tools for manipulating files that can usually construct a fake capture, which in this case wireshark or tshark main can then decrypt.