SSL/TLS handshake failure

Question

Configuration

A Windows 2008R2 RDS server RDS01
A Windows 2008R2 RDS server RDS02
A Windows 2012R2 RemoteApp server APP01

Problem

Calling URL https://APP01.domain.local/rdweb :

From RDS01 : Schannel error 40 (handshake_failure)

From RDS02 : Response code 200, cert chain OK

Diagnose

Using ssldiagnos.exe from RDS01 (handshake failure removed):

-= SUMMARY =-
Weak:         0
Intermediate: 0
Strong:       0

APP01 accepts 0 SSL handshake configurations

Using ssldiagnos.exe from RDS02 (handshake failure removed):

[+] Testing STRONG: TLS 1, ECDHE-RSA-AES256-SHA (256 bits) ... CONNECT_OK SEND_OK RECEIVE_OK CERT_OK
[+] Testing STRONG: TLS 1, DHE-RSA-AES256-SHA (256 bits) ... CONNECT_OK SEND_OK RECEIVE_OK CERT_OK
[+] Testing STRONG: TLS 1, ECDHE-RSA-AES128-SHA (128 bits) ... CONNECT_OK SEND_OK RECEIVE_OK CERT_OK
[+] Testing STRONG: TLS 1, DHE-RSA-AES128-SHA (128 bits) ... CONNECT_OK SEND_OK RECEIVE_OK CERT_OK
[+] Testing STRONG: TLS 1.1, ECDHE-RSA-AES256-SHA (256 bits) ... CONNECT_OK SEND_OK RECEIVE_OK CERT_OK
[+] Testing STRONG: TLS 1.1, DHE-RSA-AES256-SHA (256 bits) ... CONNECT_OK SEND_OK RECEIVE_OK CERT_OK
[+] Testing STRONG: TLS 1.1, ECDHE-RSA-AES128-SHA (128 bits) ... CONNECT_OK SEND_OK RECEIVE_OK CERT_OK
[+] Testing STRONG: TLS 1.2, ECDHE-RSA-AES256-SHA384 (256 bits) ... CONNECT_OK SEND_OK RECEIVE_OK CERT_OK
[+] Testing STRONG: TLS 1.2, ECDHE-RSA-AES256-SHA (256 bits) ... CONNECT_OK SEND_OK RECEIVE_OK CERT_OK
[+] Testing STRONG: TLS 1.2, DHE-RSA-AES256-GCM-SHA384 (256 bits) ... CONNECT_OK SEND_OK RECEIVE_OK CERT_OK
[+] Testing STRONG: TLS 1.2, DHE-RSA-AES256-SHA (256 bits) ... CONNECT_OK SEND_OK RECEIVE_OK CERT_OK
[+] Testing STRONG: TLS 1.2, ECDHE-RSA-AES128-SHA256 (128 bits) ... CONNECT_OK SEND_OK RECEIVE_OK CERT_OK
[+] Testing STRONG: TLS 1.2, ECDHE-RSA-AES128-SHA (128 bits) ... CONNECT_OK SEND_OK RECEIVE_OK CERT_OK
[+] Testing STRONG: TLS 1.2, DHE-RSA-AES128-GCM-SHA256 (128 bits) ... CONNECT_OK SEND_OK RECEIVE_OK CERT_OK
[+] Testing STRONG: TLS 1.2, DHE-RSA-AES128-SHA (128 bits) ... CONNECT_OK SEND_OK RECEIVE_OK CERT_OK

-= SUMMARY =-
Weak:         0
Intermediate: 0
Strong:       15

APP01 accepts 15 SSL handshake configurations

Schannel

RDS01 SSL options activation
RDS02 SSL options activation

Help

Any help to solve this issue will be welcome

Edit 1

Here, the RDS01 SCHANNEL registry hive export after enabling all (except SSL 2.0) manually with IIS Crypto to explicitly declare them as enabled

{schannel}=HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\SecurityProviders\SCHANNEL

[{schannel}]
"EventLogging"=dword:00000001

[{schannel}\Ciphers]

[{schannel}\Ciphers\AES 128/128]
"Enabled"=dword:ffffffff

[{schannel}\Ciphers\AES 256/256]
"Enabled"=dword:ffffffff

[{schannel}\Ciphers\DES 56/56]
"Enabled"=dword:ffffffff

[{schannel}\Ciphers\NULL]
"Enabled"=dword:ffffffff

[{schannel}\Ciphers\RC2 128/128]
"Enabled"=dword:ffffffff

[{schannel}\Ciphers\RC2 40/128]
"Enabled"=dword:ffffffff

[{schannel}\Ciphers\RC2 56/128]
"Enabled"=dword:ffffffff

[{schannel}\Ciphers\RC4 128/128]
"Enabled"=dword:ffffffff

[{schannel}\Ciphers\RC4 40/128]
"Enabled"=dword:ffffffff

[{schannel}\Ciphers\RC4 56/128]
"Enabled"=dword:ffffffff

[{schannel}\Ciphers\RC4 64/128]
"Enabled"=dword:ffffffff

[{schannel}\Ciphers\Triple DES 168]
"Enabled"=dword:ffffffff

[{schannel}\CipherSuites]

[{schannel}\Hashes]

[{schannel}\Hashes\MD5]
"Enabled"=dword:ffffffff

[{schannel}\Hashes\SHA]
"Enabled"=dword:ffffffff

[{schannel}\Hashes\SHA256]
"Enabled"=dword:ffffffff

[{schannel}\Hashes\SHA384]
"Enabled"=dword:ffffffff

[{schannel}\Hashes\SHA512]
"Enabled"=dword:ffffffff

[{schannel}\KeyExchangeAlgorithms]

[{schannel}\KeyExchangeAlgorithms\Diffie-Hellman]
"Enabled"=dword:ffffffff

[{schannel}\KeyExchangeAlgorithms\ECDH]
"Enabled"=dword:ffffffff

[{schannel}\KeyExchangeAlgorithms\PKCS]
"Enabled"=dword:ffffffff

[{schannel}\Protocols]

[{schannel}\Protocols\Multi-Protocol Unified Hello]

[{schannel}\Protocols\Multi-Protocol Unified Hello\Client]
"Enabled"=dword:ffffffff
"DisabledByDefault"=dword:00000000

[{schannel}\Protocols\Multi-Protocol Unified Hello\Server]
"Enabled"=dword:ffffffff
"DisabledByDefault"=dword:00000000

[{schannel}\Protocols\PCT 1.0]

[{schannel}\Protocols\PCT 1.0\Client]
"Enabled"=dword:ffffffff
"DisabledByDefault"=dword:00000000

[{schannel}\Protocols\PCT 1.0\Server]
"Enabled"=dword:ffffffff
"DisabledByDefault"=dword:00000000

[{schannel}\Protocols\SSL 2.0]

[{schannel}\Protocols\SSL 2.0\Client]
"DisabledByDefault"=dword:00000001
"Enabled"=dword:00000000

[{schannel}\Protocols\SSL 2.0\Server]
"Enabled"=dword:00000000
"DisabledByDefault"=dword:00000001

[{schannel}\Protocols\SSL 3.0]

[{schannel}\Protocols\SSL 3.0\Client]
"Enabled"=dword:ffffffff
"DisabledByDefault"=dword:00000000

[{schannel}\Protocols\SSL 3.0\Server]
"Enabled"=dword:ffffffff
"DisabledByDefault"=dword:00000000

[{schannel}\Protocols\TLS 1.0]

[{schannel}\Protocols\TLS 1.0\Client]
"Enabled"=dword:ffffffff
"DisabledByDefault"=dword:00000000

[{schannel}\Protocols\TLS 1.0\Server]
"Enabled"=dword:ffffffff
"DisabledByDefault"=dword:00000000

[{schannel}\Protocols\TLS 1.1]

[{schannel}\Protocols\TLS 1.1\Client]
"Enabled"=dword:ffffffff
"DisabledByDefault"=dword:00000000

[{schannel}\Protocols\TLS 1.1\Server]
"Enabled"=dword:ffffffff
"DisabledByDefault"=dword:00000000

[{schannel}\Protocols\TLS 1.2]

[{schannel}\Protocols\TLS 1.2\Client]
"Enabled"=dword:ffffffff
"DisabledByDefault"=dword:00000000

[{schannel}\Protocols\TLS 1.2\Server]
"Enabled"=dword:ffffffff
"DisabledByDefault"=dword:00000000

Is a Windows update installed patch set the same at RDS01 and RDS02? — Vadim, Oct 08 '18 at 17:30
Changes in schannel do not apply until the whole system is rebooted. Have you tried that? Alos, is ciphersuites branch empty on both machines? — NuTTyX, Oct 08 '18 at 21:34

Tim Brigham · Answer 1 · 2018-10-08T19:28:35.093

2

I've been bitten by setting the client side channels in a non backwards compatible fashion in the past. You probably should look at the client side protocols on RDS01.

In your situation I'd use IIS Crypto to enable all client side protocols on RDS01 (temporarily) by checking all the boxes, and using the "set client side protocol" settings, then apply best practices without the "set client side protocol".

That should rule out any client side config issues.

EDIT:

If that doesn't work I'd suggest using Wireshark to capture and dissect the handshakes and see exactly what cryptographic suites are being presented by RDS01. Personally I'd run it on APP01 so I could compare a good handshake to bad easily.

edited Oct 08 '18 at 19:28

answered Oct 08 '18 at 16:48

Tim Brigham

15,465
7
72
113

ISS Crypto screenshots was the simplest way I found to show all client side protocols are activated but that was a bad idea because it shows server side activation. Anyway, I manually activated all protocols/ciphers/hashes/key exchanges excepting SSL 2.0 to explicitly declare their activation in registry, and I restarted `RDS01` server but no change... – alex Oct 08 '18 at 17:46
`Edit 1` shows the state of `SCHANNEL` registry hive – alex Oct 08 '18 at 18:05

alex · Answer 2 · 2018-10-09T00:24:26.293

Debugging with WireShark gave me a lot of TCP fragmentation and retransmission. After adjusting router MSS setting, SSL handshakes are correctly established.

My apologies for incomplete question, the RDS01 and RDS02 servers are on the same network linked to APP01 network via an IPSec VPN gateway. Since RDS02 was working, I did not consider this possibility.

To solve my problem, I used this table and found MMS = 1384

  1500   Standard MTU
-   20   IP Header
-   24   GRE Encaps.
-   52   IPSec Encaps.
-  (8)   PPPoE (optional)
-   20   TCP Header
-------
= 1384   MMS

Open router admin page and set the new MTU or MSS