Client certificate for 2 way SSL works in some programs, and fails in most

Question

I am trying to establish 2 way SSL connection with remote server and I have received certificate to connect. Can somebody explain behaviour described below.

I am able only to use it with SOAP UI which uses Java SSL libraries.

However all other software such as browsers and .Net applications fail.

Chrome returns this error: ERR_SSL_CLIENT_AUTH_SIGNATURE_FAILED .

Firefox returns: Certificate contains unknown critical extension. (Error code: sec_error_unknown_critical_extension)

.Net app fails when trying to send client certificate.

In event log (on client side) I get audit failure with ID 5061 in both for .Net and Chrome:

Subject:
    Security ID:        IIS APPPOOL\ASP.NET v4.0
    Account Name:       ASP.NET v4.0
    Account Domain:     IIS APPPOOL
    Logon ID:       0x34e817

Cryptographic Parameters:
    Provider Name:  Microsoft Software Key Storage Provider
    Algorithm Name: RSA
    Key Name:   Key Name
    Key Type:   Machine key.

Cryptographic Operation:
    Operation:  Sign hash.
    Return Code:    0x80090028

Return code 0x80090028 translates to NTE_BUFFER_TOO_SMALL (from MSDN)

I have also attempted to start my own server (IIS) and send this certificate to server. I sent two certificates signed by same CA, and for other one it worked fine and for the problematic certificate it returned:

HTTP Error 403.7 - Forbidden

The page you are attempting to access requires your browser to have a Secure Sockets Layer (SSL) client certificate that the Web server recognizes.

EDIT: I tried to access WAS server on Windows which also requires 2way SSL.

In this case communication is established when I send certificate from file, but fails the same way when loaded from the store. Original server returns handshake failure when certificate is sent from file (some linux configuration, but I do not know the details).

EDIT2: Lines from network.log after trying to load certificate from store: Network log contains following lines:

System.Net Information: 0 : [11620] SecureChannel#22469803 - Certificate is of type X509Certificate2 and contains the private key.
System.Net Information: 0 : [11620] AcquireCredentialsHandle(package = Microsoft Unified Security Protocol Provider, intent  = Outbound, scc     = System.Net.SecureCredential)
System.Net Information: 0 : [11620] InitializeSecurityContext(credential = System.Net.SafeFreeCredential_SECURITY, context = 5455730:20410a0, targetName = <target_host>, inFlags = ReplayDetect, SequenceDetect, Confidentiality, AllocateMemory, InitManualCredValidation)
System.Net Information: 0 : [11620] InitializeSecurityContext(In-Buffers count=2, Out-Buffer length=0, returned code=InternalError).

Answer 1

The Firefox error is likely the key to the issue. It looks like the cert you have been given is tied to the API implementation using a custom critical extension. The x509 spec requires users of a cert to fail the operation if they encounter an extension marked critical that they do not understand. More info in rfc 5280

Look at the problem cert with openssl x509 -in certfile.pem -text and look for a custom extension that is marked critical.

All the other failures you are seeing are just those systems trying to cope with the required error from the spec.

Answer 2

The client certificate that you use contains something which is probably sort-of standard but irks your server in some way. What you see with your own IIS is an indirect symptom: your client sent the certificate, but IIS did not accept it, and therefore decided to continue the connection as if the client had not sent any certificate; and, for such client, your IIS is configured to reject the connection with a 403 HTTP error.

In what you describe, you do not state clearly whether the audit failure with ID 5601 is observed on the client or on the server machine; this matters because it would tell whether the client actually sent the certificate (and it was rejected by the server), or whether the client code found the certificate to be unfit in some way.

---

To investigate the problem, I suggest that you run some network monitor tool to see whether the client actually sends the certificate or not to the server. For instance, Microsoft's aptly named Network Monitor. When connecting to an IIS server which requests (or requires) a client certificate, you should observe the following:

• Client connects and sends its ClientHello.
• The server responds with a ServerHello, then Certificate and ServerHelloDone. The server does not yet ask for a client certificate at this point.
• The handshake continues with the ClientKeyExchange, ChangeCipherSpec and Finished from the client. The Finished message shows up as "encrypted handshake" because after the ChangeCipherSpec from the client, everything the client sends is encrypted.
• Server responds with ChangeCipherSpec and its own Finished message (again, "encrypted handshake").
• Client sends "encrypted application data": this is the HTTP request from the client.
• At that point, the server learns the target path, and sees in its configuration that for that path, a client certificate should be requested. The server then sends back an HelloRequest that you will see as a small "encrypted handshake" (exact size depends on the negotiated cipher suite, but you would see a size of 32 bytes with AES/SHA-1).
• Client responds with an "encrypted handshake" which contains, again, a ClientHello message; its size is about 20-30 bytes more than the previous ClientHello because of the slight overhead of encryption+MAC.
• Server sends back an "encrypted handshake" which should be substantially larger than what it sent in the first handshake, because that one will contain an extra CertificateRequest message which contains the list of names of root CA that the server uses.
• Then the client responds with an "encrypted handshake" message. This is the one you want to check. If the client does not send a certificate, then that message will be 20 to 30 bytes longer than the ClientKeyExchange sent in the first handshake, for a total length well below one kilobyte. On the other hand, if the client sends a certificate, then the "encrypted handshake" message sent by the client at that point will be quite longer, several kilobytes at least (if you master the intricate details of SSL, you can even work out the exact length of the certificate sent by the client, which can be handy if the client has access to several).
• The message from the client is followed by a ChangeCipherSpec and again an "encrypted handshake" (the Finished message) from the client.
• The server responds with a ChangeCipherSpec and an "encrypted handshake" (the Finished).
• The server sends some "encrypted application data", which is the 403 HTTP response that you observe.

Network Monitor cannot intercept local communications, so you will have to run the client on another machine.

Depending on whether the client sends the certificate or not, this will lead you to distinct investigation paths. If the client does not send the certificate, then either it does not have access to the private key, or it deems the private key unsuitable in some way (for instance, the certificate contains a Key Usage extension which marks it as "encrypt only", whereas the client must technically use the private key for a signature); or maybe the client does not know enough about the certificate chain to decide that the certificate indeed links up to one of the root CA that the server accepts. Another possibility is that the key pair (public key is in the certificate) exceeds some internal limit. Microsoft software will have trouble with RSA keys shorter than 1024 bits, or longer than about 4096 bits (2048 bits would be a safe length). Microsoft software also has trouble if the key's public exponent does not fit on 32 bits.

Most RSA keys use 65537 as public exponent (a 17-bit integer) but I have encountered other values. Java's implementation is Java-based (duh...) and it has no problem with arbitrarily long public exponents, which would explain the symptoms.

If the client indeed sends the certificate to the server, then this removes the private key from the equation (the client could obviously use it). In that case, "opening" the certificate on the server (export the certificate without its private key, as a ".cer file", put the file on the server, double-click on it) will show you what Windows thinks of that certificate; if it is "bad" in some way this will be displayed graphically.

(The paragraphs above describe what I do half of the time at my job. That's not my theoretical function, but in practice I "solve problems", and most problems I solve are about certificates and SSL, and resolution almost always includes Network Monitor to understand what the heck really happens. Error messages from browsers or from IIS or events in the audit logs are, as a rule, thoroughly unhelpful. Knowledge of the details of SSL helps a lot; see this answer for an introduction if you want to learn about it.)