Debugging client certificate issues on IIS

Question

We have an 2008R2 IIS server set up with a site configured to require client certificates. Our test client isn't working, and we're trying to debug why.

During the course of this, we've set up a new Server 2008 R2 box (yes, I know it's old, but this is what's running the software) to try and replicate or determine ways to troubleshoot.

One route we're investigating is the TLS handshake. The test application is written in .NET and with appropriate System.Diagnostics debugging enabled, this places the following entry in a log file:

System.Net Information: 0 : [22724] SecureChannel#48979325 - We have user-provided certificates. The server has specified 10 issuer(s). Looking for certificates that match any of the issuers.

We're unable to see this list of issuers, so we broke out OpenSSL. Running the following command:

openssl s_client -connect win2k8r2-1.hsl10690.test:443 -state -no_ticket -servername win2k8r2-1.hsl10690.test

Resulted in output that stated:

   [...]
-----END CERTIFICATE-----
subject=/CN=testcert.hsl10690.test
issuer=/CN=Internal Dev CA 1
---
No client certificate CA names sent
---
SSL handshake has read 1013 bytes and written 329 bytes
   [...]

So we have a mismatch where the Microsoft stack declared that the server specified 10 issuers, but OpenSSL is reporting that the server sent no CA names.

In the case of the live system, the System.Diagnostics logs reports 130-odd issuers specified by the server, yet OpenSSL still returns zero.

We believe the issue is that the client certificate we're supplying doesn't match one of the issuers (but we've validated that the root is in the server's trust store, and we've validated the certificate outside of the server). On the live server we see this in the logs after the "The server has specified..." message:

System.Net Information: 0 : [36484] SecureChannel#33675143 - We have user-provided certificates. The server has specified 133 issuer(s). Looking for certificates that match any of the issuers.
    ProcessId=20372
    DateTime=2018-12-20T13:33:39.9042036Z
System.Net Information: 0 : [36484] SecureChannel#33675143 - Left with 0 client certificates to choose from.
    ProcessId=20372
    DateTime=2018-12-20T13:33:39.9052036Z

whilst on test, where things work, it says:

System.Net Information: 0 : [22724] SecureChannel#48979325 - We have user-provided certificates. The server has specified 10 issuer(s). Looking for certificates that match any of the issuers.
    ProcessId=22100
    DateTime=2018-12-21T13:52:23.3718249Z
System.Net Information: 0 : [22724] SecureChannel#48979325 - Selected certificate: [Version]
  V3

[Subject]

How can we find out what certificates are returned by the server, and if we find that the issuer is missing from the list, what could have prevented the root being included? I'm not ruling out that we've missed something obvious, but we've not seen it yet.

Answer 1

We finally found the answer to this: at fault is KB931125. A blog article describes the KB as only intended for client targets, but got shipped to servers which results in the CA list being sent by IIS to be truncated. It doesn't really explain why we couldn't see the transmitted CA list with OpenSSL, but ultimately it did get us to the root cause.

The blog article points to a further MSDN article that describes the issue in more detail:

These problems may occur if a TLS/SSL server contains many entries in the trusted root certification list. The server sends a list of trusted certificate authorities to the client if the following conditions are true:

• The server uses the Transport Layer Security (TLS)/SSL protocol to encrypt network traffic.
• Client certificates are required for authentication during the authentication handshake process.

This list of trusted certificate authorities represents the authorities from which the server can accept a client certificate. To be authenticated by the server, the client must have a certificate that is present in the chain of certificates to a root certificate from the server's list. This is because the client certificate is always the end-entity certificate at the end of the chain. The client certificate isn't part of the chain.

Currently, the maximum size of the trusted certificate authorities list that the Schannel security package supports is 16 KB in Windows Server 2008, Windows Server 2008 R2, and Windows Server 2012.

An event log entry gets logged as well that says (which we missed the first time we looked at this):

When asking for client authentication, this server sends a list of trusted certificate authorities to the client. The client uses this list to choose a client certificate that is trusted by the server. Currently, this server trusts so many certificate authorities that the list has grown too long. This list has thus been truncated. The administrator of this machine should review the certificate authorities trusted for client authentication and remove those that do not really need to be trusted.

The MSDN article does list a fix:

Delete the following registry key:

HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\SystemCertificates\AuthRoot\Certificates

To do this, follow these steps:

1. Start Registry Editor
2. Locate the following registry subkey: HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\SystemCertificates\AuthRoot
3. Right-click and then delete the key that is called "Certificates"