SSL handshake failure modes

Question

In SSL if the handshake is not successful, does it always end with a handshake alert?
Or are there other ways to finish the SSL connection (acceptable by standard).
I am asking this, because in an HTTPS server configured to require client authentication, if the client does not send a certificate, I see (via network packets) a TLS Finished message from server to client; I was expecting an alert.

Answer 1

A SSL/TLS connection must always end with an alert message of some kind: a connection which is broken (e.g. the underlying TCP socket is closed) without an alert is said to be improperly terminated. The alert message for a normal termination is close_notify.

The handshake is only a part of the connection; a handshake occurs at the start and subsequent handshakes can be played on the same connection. The handshake is usually where errors occur, triggering alert messages, but a fatal alert message does not end only the handshake: it closes the whole connection.

In the case of a server requesting client authentication, the client may respond by sending a certificate and computing a signature; however, this is optional. The client is allowed not to honor the request. The server then chooses what to do. The server may reject the client right away, through an alert message which closes the connection. The server may also decide to continue, and handle the lack of client authentication at another level. A server which sends a Finished message certainly chooses to continue; possibly, the server may decide to report the error as applicative data within the tunnel (e.g., for a HTTPS server, send an HTTP error message once the handshake is finished).

See RFC 2246, section 7.4.6 for details:

   This is the first message the client can send after receiving a
   server hello done message. This message is only sent if the
   server requests a certificate. If no suitable certificate is
   available, the client should send a certificate message
   containing no certificates. If client authentication is required
   by the server for the handshake to continue, it may respond with
   a fatal handshake failure alert.

→ emphasis on "may".

Answer 2

The section on client certificates in the TLS 1.0 specification (RFC 2246) says this:

If no suitable certificate is available, the client should send a certificate message containing no certificates. If client authentication is required by the server for the handshake to continue, it may respond with a fatal handshake failure alert.

The fact that it may respond with a fatal alert doesn't mean that it has to.

You've tagged your question with IIS, but here is a bit of background about other servers. In Apache Httpd, there are three settings for requesting a client certificate: none, optional and required (and optional_no_ca if you want to disable the certificate verification there). When the server is configured with optional, when the client doesn't present a certificate, the handshake will still proceed. When the server is configured with required, if there's no client certificate, there will be a fatal handshake failure alert. Java's SSLEngine implements these settings with setWantClientAuth and setNeedClientAuth.

Note that proceeding with the handshake doesn't mean that the server will grant access to the resource or authorize the execution of the request.

The big downside of the fatal handshake failure alert behaviour is that it closes the connection abruptly. At best, it will appear as a "connection closed - handshake failure" message (or something similar) in the browser. No HTTP exchange will have taken place at all, so there is no opportunity to send an HTTP status code and/or an accompanying web page to describe the problem.

From a usability point of view, this can be very confusing, especially for users who are not necessarily familiar with client certificates (there is often a certain learning curve simply for dealing with certificates). For this reason, many servers only require optional client-certificate authentication, so as to be able to send an explanation if authorization is denied.

I must admit I'm not entirely familiar with the settings in IIS, but the documentation seems to suggest there are three options too:

Select Ignore if you do not want to accept a client certificate even if a client presents one.

Select Accept to accept client certificates.

Select Require to require client certificates. To use Require Client Certificates, you must enable Require SSL.

Another document suggests there is an HTTP status code and error message for failing to provide a required certificate: "403.7 Forbidden: Client certificate required". The fact that there is an HTTP status code for this case implies that it's not a case that causes a handshake failure (otherwise the connection wouldn't be established to send HTTP messages at all). This implies that the "require" mode of IIS behaves like the "optional" mode of Apache Httpd as far as the TLS handshake is concerned, that is, not presenting a client certificate doesn't cause a fatal alert (authorization afterwards is a different matter, of course).

I'm not sure what "ignore" and "accept" are for, then. In SSL/TLS, only the server can request a certificate, by sending a CertificateRequest message. If the server doesn't send this message, the client will simply never send its certificate, again, from RFC 2246:

7.4.6. Client certificate

When this message will be sent: This is the first message the client can send after receiving a server hello done message. This message is only sent if the server requests a certificate.

My guess is that they mean "ignore"/"accept" in terms of account mapping or authorization, further down the line.

EDIT.

As far as I remember, by default, IIS always negotiates client-certificates using re-negotiation: a first handshake is successful, without any client-certificate request, but then, a second handshake is triggered. This means that you wouldn't be able to see the client-certificate exchange by looking at the packets, since this second handshake would be done within the encrypted session (you could see it with Wireshark in some cases, it it's configured with the server's private key and it supports the cipher suite).

You can configure the client-certificate negotiation to be done within the initial handshake using the clientcertnegotiation=enable option of netsh (which refers to the initial handshake).

Answer 3

I will take an example from OpenSSL, which is used by Apache.

There is a mode flag, called SSL_VERIFY_PEER which when set on the server, it will ask for a client certificate at the handshake. The client is free to send it or not.

Then the behavior of the server, depends on another flag:

SSL_VERIFY_FAIL_IF_NO_PEER_CERT

Which is explained as follows

Server mode: if the client did not return a certificate, the TLS/SSL handshake is immediately terminated with a ``handshake failure'' alert. This flag must be used together with SSL_VERIFY_PEER.

If this flag is not set, no alert is created and the handshake goes on as normal and the server has to check for the certificate afterwards and close the connection if necessary.

EDIT: More information about similar configuration options in IIS 7.0:

https://www.iis.net/ConfigReference/system.webServer/security/access

It seems that there is an sslFlags attribute which can be one of the following

• None. This default setting disables SSL for the site or application.
• Ssl. The site or application requires SSL.
• SslNegotiateCert. The site or application accepts client certificates for authentication.
• SslRequireCert. The site or application requires client certificates for authentication.
• Ssl128. The site or application requires 128-bit SSL certificate encryption.

I imagine that the behaviour you notice happens because the SslNegotiateCert is configured instead of SslRequireCert.