SSL failure on Auzre web farm with shared configuration and centralized certificates

Question

[Sorry if this question is a bit long, there's lot of extra info in case it is relavent]

Overview

I'm having a problem with SSL on a 4 VM's load balanced farm in Azure. If the HTTPS request goes to the first server in the farm, all is well. If it arrives at any of the other 3, then the call fails. Chrome will issue an SSL protocol error, IE and Firefox simply say the page can't be displayed.

The Setup

I have four Windows Server 2012 VM's on Azure (Small instance as just testing). The VM's are in the same cloud service and availability set. Load balanced endpoints have been added for ports 80 and 443 (direct server return is NOT enabled on these). All four machines were set up via the same PowerShell script and with two exceptions were fully configured this way.

The IIS configuration of each server is set to use a share configuration, which is replicated via DFS to each server from the first server in the group. This was manually configured for each server and is working fine.

DFS is also used to replicate the webs folder from the first server to the others.

I also discovered the "new" Centralized Certificates feature after writing the deployment script, so this was manually installed and configured on the four servers too. I'm using a share on a separate server to store the certificate files.

The Certificate Request was generated on the first server, and I used SSL.com to get a free 90 day SSL for an address subdomain.domain.com. I added a CNAME record for subdomain to the DNS for domain.com pointing to the Azure cloudapp address.

I imported the SSL certificate into the first server, then exported it again as subdomain.domain.com.pfx (with a password) and copied that into the certificate files share. When checking Centralized Certificates on all four servers, they are listing the certificate fine with no error icons indicating the password on the config was wrong etc.

Finally, I changed the bindings of server 1 to add https with host name subdomain.domain.com, and with the Require Server Name Indication and Use Centralized Certificate Store options checked. Checking the other servers shows the bindings propagated as expected.

The Problem

I added a basic page which simply spits out the name of the server the request was handled by. If I shell several IE windows accessing http://subdomain.domain.com, they'll print a variety of server names, showing that the IIS config and web files are being deployed correctly, and the Azure loadbalancing thingy is doing it's stuff too. Which I found really cool actually.

However, it goes down the drain when I try this via HTTPS. Only the requests that hit the first server succeed, the rest of them crash and burn with "This page can't be displayed" or "SSL Protocol Error" depending on which browser I test with. On server 1, the page displays fine, the certificate is viewable and there are no certificate errors.

I'm sure this is a configuration problem on the servers somewhere, but I just can't tell what on earth it is. Most of what I'm playing with is new to me as in the past we've used physical non-clustered Windows 2003 Servers with IIS6.

What is even more perplexing is I shutdown the four VM's overnight, and when I restarted the service this morning, server 2 now apparently responds to SSL requests in addition to server 1. With that said though, I had done a full shutdown yesterday too and still only server 1 worked after that.

The IIS log files don't show the failed SSL requests, only a bunch of 200's for port 80 and a mix of 200 and 304 for port 443 on server 1 and 2.

I have done some due diligence with Google searches but nothing is shedding any light. The exact opposite in fact, from what I can tell what I've done should just work.

Any advice to help resolve this would be gratefully received.

Answer 1

I wanted to share some steps that should resolve most issues with the Centralized Certificate Store on IIS. The Centralized Certificate Store (CCS) creates a faux certificate binding which it uses to funnel SSL certificate requests through to the CCS. If this binding is not present or there are multiple bindings on the same IP address, you will receive browser errors when clients attempt to connect to SSL enabled websites.

Consider the following setup; an IIS server with IP address 172.16.0.41 and Centralized Certificate Store enabled contains two domains; www.adomain.com and www.bdomain.com. PFX certificate packages are installed in the IIS Centralized Certificate Store for each of the domains.

There are typically two errors;

PROBLEM 1

When browsers connect to the website, the wrong certificate is shared. For example visiting www.adomain.com the first time results in the correct certificate being served however when a browser visits www.bdomain.com, the certificate for www.adomain.com is returned, causing the browser to flag an invalid certificate error.

SOLUTION

The reason for this issue is usually because multiple websites are bound to the same IP address and at least one of the websites has not enabled Require Server Name Identification. Without this setting enabled for all web sites sharing the same IP and PORT, IIS is unable to determine which certificate to serve and so a first one in wins policy appears to apply.

This means subsequent requests to other websites hosted on the same IIS server return the wrong certificate.

Steps to resolve

1. Make sure each website is set to Require Server Name Identification. Do this by clicking on each website in IIS Manager, choosing Bindings...->Select the HTTPS Binding->Choose Edit->Check Require Server Name Identification and ensure Use Centralized Certificate Store is also checked.

2. Check the bindings on the server. From an elevated command box execute netsh http show sslcert

The binding for the CCS should be present:

SSL Certificate bindings:
------------------------- 

Central Certificate Store    : 443
Certificate Hash             : (null)
Application ID               : {4dc3e181-e14b-4a21-b022-59fc669b0914}
Certificate Store Name       : (null)
Verify Client Certificate Revocation : Enabled
Verify Revocation Using Cached Client Certificate Only : Disabled
Usage Check                  : Enabled
Revocation Freshness Time    : 0
URL Retrieval Timeout        : 0
Ctl Identifier               : (null)
Ctl Store Name               : (null)
DS Mapper Usage              : Disabled
Negotiate Client Certificate : Disabled

Note the binding with the (null) Certificate Hash indicates the presence of the CCS pass through binding. If there are any other bindings listening on your server's IP address and the SSL port (172.16.0.41:443 in our scenario), these need to be removed.

2. To remove the bindings, type netsh http delete sslcert <binding>. In our scenario this would be netsh http delete sslcert 172.16.0.41:443 and the binding should be removed. The (null) CCS passthrough binding should not be removed. If this binding is not present, see PROBLEM 2 below.

3. Reset iis with iisreset and connect to each domain on the web server. The correct certificate should be shared. You can also check the bindings by repeating step 2 and ensuring no specific IP:PORT bindings to your SSL port and web server IP exist.

PROBLEM 2

Even after confirming the Centralized Certificate Store is enabled, has visibility on the certificates and all websites are bound to use it, when a browser connects to the website it throws a Page Cannot be displayed or invalid encryption error. The intended website does not display.

In IE (Edge) this will typically manifest itself as a suggestion to check that appropriate encryption types are enabled in the browser.

An iisreset or reboot of the server does not resolve the issue.

SOLUTION

This appears to be a bug in the creation of the CCS pass-through binding. It is not created on the web-server when the CCS functionality is enabled so that incoming requests for a certificate are silently dropped.

Steps to resolve

1. First check to that Centralized Certificate Store has been enabled and can see all the certificates in its path. To do this, check IISManager choose the server node->Centralized Certificates (under Management section)->Edit Feature Settings.. and ensure its configured and enabled.

2. Check that all the websites that use common certificates are bound to the CCS. Do this by clicking on each website in IIS Manager, choosing Bindings...->Select the HTTPS Binding->Choose Edit->Check Require Server Name Identification and ensure Use Centralized Certificate Store is also checked.

3. Check the CCS binding has been created. Run netsh http show sslcert. If the results are empty or the (null) pass through certificate is not present (see step 2 under PROBLEM 1 for what the CCS binding looks like), the CCS binding has not been enabled.

   SSL Certificate bindings:
   -------------------------
   

4. In IIS Manager, choosing one website that uses the CCS, click Bindings...->Select the HTTPS Binding->Choose Edit->and uncheck Use Centralized Certificate Store and uncheck Require Server Name Identification.

In the SSL certificate drop-down, select the servers default WMSVC certificate and click OK. Leave the Site Bindings window open.

5. Switch back to the elevated command prompt and run netsh http show sslcert. This should yield a binding like so:

   SSL Certificate bindings:
   ------------------------- 
   IP:port                      : 172.16.0.41:443
   Certificate Hash             : 64498c920fecb31b8f7ccbdac2fa2baa2ec4f19a
   Application ID               : {4dc3e181-e14b-4a21-b022-59fc669b0914}
   Certificate Store Name       : My
   Verify Client Certificate Revocation : Enabled
   Verify Revocation Using Cached Client Certificate Only : Disabled
   Usage Check                  : Enabled
   Revocation Freshness Time    : 0
   URL Retrieval Timeout        : 0
   Ctl Identifier               : (null)
   Ctl Store Name               : (null)
   DS Mapper Usage              : Disabled
   Negotiate Client Certificate : Disabled
   

6. Return to the Site Bindings window and revert the changes from Step 4. Enabling Require Server Name Indication and Use Centralized Certificate Store and click OK.

7. Switch back to the elevated command prompt and run netsh http show sslcert once again and - if the gods smile upon you - the Centralized Certificate Store binding should have burst into existence, replacing the current binding:

     SSL Certificate bindings:
     -------------------------
   
     Central Certificate Store    : 443
     Certificate Hash             : (null)
     Application ID               : {4dc3e181-e14b-4a21-b022-59fc669b0914}
     Certificate Store Name       : (null)
     Verify Client Certificate Revocation : Enabled
     Verify Revocation Using Cached Client Certificate Only : Disabled
     Usage Check                  : Enabled
     Revocation Freshness Time    : 0
     URL Retrieval Timeout        : 0
    Ctl Identifier               : (null)
    Ctl Store Name               : (null)
    DS Mapper Usage              : Disabled
    Negotiate Client Certificate : Disabled
   

If you see the above certificate binding with the (null) valued Certificate Hash it has worked and the CSS pass through should now be enabled.

8. Fire up a browser and attempt to connect to the domains on the server over the secure socket (HTTPS) and all should be well.

Important notes

• You need to repeat this process on each web server in your web-farm. It should be doable via PowerShell to automate and check the process.

• Once the passthrough binding has been created, it will be retained indefinitely (unless you disable Centralized Certificate support on the node).

As an aside; this link has information on how the CCS works at a technical level and is worth a read: https://blogs.msdn.microsoft.com/kaushal/2012/10/11/central-certificate-store-ccs-with-iis-8-windows-server-2012/

I hope that helps.