I'm trying to intercept a SSLv3 connection between a PlayStation 3 client and an authentication server of and outdated game to reverse engineer its protocol.
I already tried to set up a simple MITM attack and copied every value from the original certificate. However, the client immediately drops the connection. I'm assuming the binary compares more than just the common fields. But it doesn't seem to validate the certificate to a CA because the PS3 does not contain the used CA (probably a custom one by the publisher) as it seems. If I open a connection via PS3's browser it throws a cert-error.
I created a broken certificate by running openssl genrsa -3 -out fake.key 512
to match the original server's cert first and replacing my public key with the public key of the original cert. Obviously this won't work because the public/private-pair is broken, nevertheless I generated the cert and signed the original cert with my private key. This resulted in a certificate with identical public keys I could use for my proxy server (PS3 -> my_fake_server -> official server).
The interesting part is that the client didn't drop the connection with this faked cert. I could see in Wireshark that it continues with the handshake (because it seems to "just" compare the public key and the common fields) and fails because of the broken pair. However, this is not the client dropping the connection but the OpenSSL implementation failing with bad_record_mac
which could indicate that the server is using a non-matching pair [1], which in fact is true.
But this makes me wonder if I could counter this error and successfully intercept the connection if I could generate a pair that is valid but contains the same public key? Could that work? How likely is it to force a collision on a 512 bit key? Are there other options I could use to succeed?