WinHTTP - Prevent successful handshake if peer certificate is invalid

Question

I am using WinHTTP on a Delphi project to make calls over to the server.

Script:

userAgent := 'TestClient.exe';
  hsession := WinHttpOpen(pwidechar(userAgent), WINHTTP_ACCESS_TYPE_DEFAULT_PROXY, nil, nil, 0);
  if(hsession = nil) then ShowMessage('Failed WinhttpOpen');
  p := 'https';
  port := 443;
  requestflags := WINHTTP_FLAG_SECURE;
  server := '10.0.0.221';

  hconnection := WinHttpConnect(hsession, PWideChar(server), port, 0);
  if(hconnection = nil) then
  begin
    le := GetLastError;
    ShowMessage('Failed to connect: ' + IntToStr(le));
  end;

  Action := 'GET';
  hInetRequest := WinHttpOpenRequest(hconnection, pwidechar(Action), nil, nil, nil, nil, WINHTTP_FLAG_SECURE);
  if(hInetRequest = nil) then
  begin
    le := GetLastError;
    ShowMessage('Failed to connect: ' + IntToStr(le));
  end;

  WinResult:=WinHttpSendRequest(hInetRequest, nil,0, 0, 0,0,0);

  if(not WinResult) then
  begin
    le := GetLastError;
    WinHttpCloseHandle(hInetRequest);
    ShowMessage('No result obtained : ' + IntToStr(le));
  end;

Question; For security compliance (FIA_X509_EXT1.1), should connection terminate right after SSL handshake? In case peer certificate is deemed invalid. Or is this ok to terminate later?

Actual: What's happening, is that the client (using WinHTTP) makes a call and successfully confirms the TLS handshake, even when the certificate is invalid. However, right after handshake and before completing request, it terminates the connection throwing a '12175' error. Which is the error for invalid certificates? Which is right?

Problem: For above (FIA_X509_EXT1.1) compliance to pass as per verification tool, WinHTTP must not allow successful handshake, and hence terminate the connection earlier.

Wireshark screen attached below: (10.0.0.221 is the server)

I already asked this question in stack exchange, but as suggested by another member, this is probably the right place.

Steffen Ullrich · Accepted Answer · 2018-02-08T06:04:54.153

Question; For security compliance (FIA_X509_EXT1.1), should connection terminate right after SSL handshake? Incase peer certificate is deemed invalid. Or is this ok to terminate later?

TL;DR: I don't think failing in TLS handshake is an actual requirement. But it will be much harder to prove secure behavior if it does not already fail in the TLS handshake.

I cannot see anything in FIA_X509_EXT1.1 and related which explicitly requires the client to already fail inside the TLS handshake. And from a security perspective it should not matter if the TLS connection fails inside the handshake or after the handshake as long as no application data get sent by the client over the untrusted connection or server data get processed. This means that as long as the certificate of the server is not fully validated:

The client should not sent any application data to the server.
The client should not process any application data sent by the server.

If the client fails the TLS handshake already in case of a bad certificate this is easy to show because application data can only be sent and processed after successfully completing the TLS handshake. But, if one allows to fail the TLS connection only after the TLS handshake then one needs to have a closer look at the specific application behavior and also the upper layer protocols.

There are protocols like FTPS, SMTPS ... where the server sends data before the client. In these cases one might actually see application data from the server in the packet capture even with bad certificates since the TLS handshake got completed successfully from the perspective of the server. Thus one needs to show somehow that the client did not process these (untrusted) data which would probably involve to dig deeper into the clients and/or TLS stacks source code.

In other protocols like HTTP the client sends data first. In these cases no application data should be seen in the packet capture at all if the certificate validation should fail. But, one would probably still need to analyze the case where a server is not conforming to the protocol and sends data to the client. Here one also needs to prove somehow that the client does not process these untrusted data from the server.

In summary: I don't think it is an actual requirement that the TLS connection should fail already inside the handshake if the certificate cannot be validated. But, if the TLS connection only fails after the TLS handshake is completed it will be much harder to verify that the no sensitive data got sent over the untrusted connection and that no untrusted data got processed by the client.

This seems to be right to me as well. Specifically, i tested this behavior on IE, Edge, Chrome and firefox. And all exhibit similar behavior. i.e. TLS handshake is successful but connection is terminated/reset later — Muhammad Uzair Khan, Feb 08 '18 at 07:20
Adding in hope it might help someone else too. Got confirmation from lab that NIAP had previously validated such appilcation that successfuly completes handshake but later terminate connection before sending data (incase of invalid certificate). This should suffice. Im marking this as accepted answer. Though NIAP's TD changes often so this might not be applicable later. But for now, this is valid — Muhammad Uzair Khan, Feb 16 '18 at 05:41

WinHTTP - Prevent successful handshake if peer certificate is invalid

1 Answers1