I'd like to host a website that should listen to subdomains (e.g. sub.domain.com) together with multiple websites that live just under a second-level domain (e.g. domain2.com, domain3.com) with IIS and with SSL.
For the website with the subdomains I have a wildcard certificate (*.domain.com) and I also have certificates specifically for the other sites (domain2.com and domain3.com).
Can such a setup be hosted on the same IIS (if that matters, in an Azure Cloud Service web role)?
The issue is exactly what titobf explained here: theoretically for this we'd need bindings using SNI, with the host specified for domain2/3.com and then a catch-all website with * host for *.domain.com. But in practice no matter how the bindings are set up if the catch-all website is on it will also receive all requests to domain2/3.com (although supposedly it's matched only as a last resort).
Any help would be appreciated.
Still unsolved
Unfortunately I wasn't able to solve this: it seems to be solvable only in extremely complicated ways, like creating a software that sits between IIS and the internet (so basically a firewall) and modifies incoming requests (before the SSL handshake takes place!) to allow the scenario. I'm fairly confident this is not possible with IIS, no matter what, not even from a native module.
I have to clarify: we use Azure Cloud Services, so we have a further constraint that we can't use multiple IP addresses (see: http://feedback.azure.com/forums/169386-cloud-services-web-and-worker-role/suggestions/1259311-multiple-ssl-and-domains-to-one-app). If you can point multiple IPs to your server then you don't have this issue since you can create bindings for IPs too, and those will work together wildcard bindings. More specifically, you need an IP for the wildcard site (but since you have a separate IP now you wouldn't have to configure a wildcard host name binding) and another IP for all other non-wildcard ones.
Actually our workaround was the usage of a non-standard SSL port, 8443. So the SNI binding is actually bound to this port, thus it works along with the other bindings. Not nice, but an acceptable workaround for us until you can use multiple IPs for web roles.
The non-working bindings now
The first https binding is SNI with a simple certificate, the second is not SNI, with a wildcard certificate.
The http site works, as well as the SNI https site, but the one with the wildcard binding gives an "HTTP Error 503. The service is unavailable." (without any further information, no Failed Request Tracing or Event Log entry).
Finally getting it basically working
Enabling the ETW trace log like Tobias described showed that the root error was the following:
Request (request ID 0xF500000080000008) rejected due to reason: UrlGroupLookupFailed.
As far as I understand this means that http.sys is not able to route the request to any available endpoint.
Checking the registered endpoints with netsh http show urlacl
showed that there was indeed something registered for port 443:
Reserved URL : https://IP:443/
User: NT AUTHORITY\NETWORK SERVICE
Listen: Yes
Delegate: No
SDDL: D:(A;;GX;;;NS)
Removing this with netsh http delete urlacl url=https://IP:443/
finally enabled my SSL binding.