How does server validate that self-signed certificate came from the client who self-signed the certificate in mutual TLS?

Question

I have a mTLS (client authentication) setup where client sends a self-signed certificate. This self-signed certificate is transferred to and trusted by the server. The server's certificate is signed by Let's encrypt and Let's encrypt CA is trusted by the client.

[ client.key ] --
                | (signed)
[ client.crt ] <-
                                                         (trusts clients.crt)

 |--------------|               client.crt                  |------------|
 |    Client    |       --------------------------->        |   Server   |
 |--------------|                                           |------------|

How is authenticity of a client is checked when the Server receives client.crt? In other words, how does a Server know client.crt is indeed came from the Client who self-signed it?

My understanding with client.crt is that its a Public Key Certificate and because it's "public", there's no security harm if it's stolen by any other parties. Is this an incorrect assumption?

It seems odd that a "public" certificate should be treated like a secret because it's self-signed and used for client authentication.

---

My guess is that when a Server receives client.crt, it would encrypt some message using client's public key included in the client.crt and have the client decrypts it. Authentic client would be able to since it owns a private key associated with the public key where as malicious client with just the client.crt wouldn't because it doesn't have the private key.

Thanks in advance!

Answer 1

The client certificate is public, just like the server certificate. The only reason to keep the client certificate would be privacy, if you don't want to reveal that you are this particular client. Publishing the client certificate does not allow anyone to impersonate you.

Client authentication does not only require the client to show that it knows the certificate. The client shows the certificate to identify itself, i.e. to claim “this is who I am”. Showing the certificate is not authentication: for authentication, the client needs to prove “this is who I am”.

The client can prove who it is because it knows the private key associated with the public key contained in the certificate. It proves that it knows the private key by making a signature of data that includes a unique challenge sent by the server. (The challenge is some random bytes sent at the very beginning of the TLS handshake.) Including a unique challenge prevents against replay attacks: if an eavesdropper records the client's responses, they won't be able to replay them to make a valid connection to the server, because next time the server's challenge will be different.

In order to decide whether the client is acceptable, the server does two things:

• The server checks that the certificate is acceptable. It may have a list of acceptable certificates (this works for self-signed certificates), or it may only accept certificates that have been signed by a particular entity. This shows that if the client is who it claims to be, it's acceptable.
• The server checks that the signature from the client is correct, using the public key contained in the certificate. This shows that the client is who it claims to be.

All of these mechanisms are similar in principle to how a client authenticates a server, although the protocol is not perfectly symmetric.

Having the sever encrypt a challenge with the client's public key, and the client prove that it knows the private key by decrypting this challenge, would also work, from a security point of view. It is not done for several reasons:

• It would require the client's private key to be capable of doing encryption. The types of private keys commonly used for TLS are RSA keys, which can do both signature and encryption, and ECDSA keys, which can only do signature.
• It would involve asymmetric decryption. Assymetric decryption is considerably harder to implement securely than signature, because it involves both the private key and externally-supplied data (the thing to decrypt) in the same operation. There is a very long history of vulnerabilities in RSA decryption. The history of RSA signature is a lot less depressing.
• It would make the protocol less efficient: it would need an extra round trip. The server can send a challenge to sign at the beginning of the connection. If it had to send a challenge to decrypt, it would first need to know the client's public key. Before the client could send its public key, it needs to know what cryptographic algorithms the server support and whether the server wants client authentication. The client sends both its certificate and its signature in the second flight of messages that it sends to the server, when it knows what algorithms the server supports. If the protocol used decryption instead of signature, the client would have to send its certificate in the second flight and there would have to be a third flight for the decrypted challenge.

Answer 2

This self-signed certificate is transferred to and trusted by the server.

and

how does a Server know client.crt is indeed came from the Client who self-signed it?

Simple: if server trusts such certificate, then a copy of this self-signed certificate is installed on a server soemwhere in the trusted root CA store and server can check if presented certificate matches any in the trust store.

Answer 3

There are a few ways that this can be done. One way is where the server maintains a list of client certificates that are authenticated. Another way is for the server to sign client certificates that are authenticated.