I read recently about next generation firewalls that use deep-packet-inspection, intrusion-prevention and something the manufacturers call encrypted-traffic-inspection, encrypted-traffic-analytics.
The manufacturers claim the encrypted traffic inspection/analysis in nextgen firewalls is made without decrypting the traffic.
Can we explain intuitively how these appliances work and what are the main points to consider from a cryptographic perspective?
The manufacturers claim the encrypted traffic inspection/analysis in nextgen firewalls is made without decrypting the traffic.
While some analysis can be done without decryption, it is fairly limited compared to full decryption. But since in some cases full decryption is not possible (privacy reasons, certificate pinning, client certificates, ... - or simply only passive analysis inside an IDS) some systems offer analysis without decryption on top of it. Such analysis is restricted to metadata contained in the initial handshake and to traffic pattern, i.e. timing, direction and size of packets.
Traffic pattern are often combined with some kind of statistics or more or less sophisticated machine learning in order to detect the kind of traffic. This can be used for example to detect DNS over TLS or HTTPS or streaming video, but can highlight also unusual traffic (small request, small response and then connection close) which might be associated with C2 communication. In general such methods have a significant false positive and negative rate, so they can be used as interesting indicators but not to retrieve 100% reliable information.
Metadata of the initial handshake can be used in heuristics to defer the kind of client (JA3 fingerprint) or server (JA3S fingerprint) based on offered or supported ciphers, protocols, TLS extensions, ... . Especially the client fingerprint is interesting, since it can often be used to distinguish normal browsers from other clients like malware traffic. This is because browsers often use a different TLS stack or different settings in the TLS stack as other clients. But there is nothing inherently magic here, so malware might also generate the same fingerprint as browsers. But if combined with traffic pattern it can result in a more reliable detection.
Inside the TLS ClientHello there is also usually the server name in clear (SNI) which can be used to compare against block lists. With ESNI though this information is slowly vanishing.
Even faster vanishing is access to the certificate. Up to TLS 1.2 the server certificate was sent in clear and it could be easily extracted. It was used in IDS/IPS for example to detect self-signed or unusual certificates which were in the past often associated with C2 communication. With TLS 1.3 the certificates are encrypted though. Also malicious sites today today also often use the free certificates offered by Let's Encrypt and others, so the value of having access to the certificate decreased.
For a deeper and much longer analysis see Encrypted Traffic Analysis - Use Cases & Security Challenges from ENISA.
These encrypted-traffic-inspection authenticate as if they are the website you are trying to visit. They can do this because they use a server certificate (and private key) signed by their own CA / PKI, of which the root certificate which is trusted by your browser / applications.
These trusted (root) certificates are generally shared through e.g. Active Directory group policies controlled by an admin, which pushes them into the Windows certificate store. This certificate store is also used by e.g. Chrome on Windows, although Google is thinking about using their own. In that case I presume some kind of service or application needs to configure the certificates in the browsers.
Finally they act as a client to the other system, operating as a man-in-the-middle, inspecting the (plaintext) data in the connection before forwarding it.
If no MitM configuration is possible then other properties may still be studied. It is possible to also inspect meta-data (as presented in the certificates within the header), the protocol used as well as side channel data such as packet / stream size and timing information. It is known that it is even possible to decode speech that way for specific codecs.
However, it is unknown to me how advanced these kind of techniques are and what their success rates is. It is also be possible for e.g. malware writers to make their communication harder to detect in response, triggering another arms race.
The encrypted traffic inspection, or TLS interception, can occur because they present a certificate that is trusted by the client. This is usually because in a corporate environment, the network administrators have installed a root certificate into the list of authorities trusted by the client. The certificate presented to the end user is generated on the fly and the client essentially sets up a TLS connection to the middlebox. The middlebox then sets up a connection to the real server and proxies the data to the real server, inspecting it, and possibly modifying it.
With a modern version of TLS (1.2 or newer), it isn't possible to intercept data unless the TLS middlebox is trusted in this way; the protocol would be insecure if it allowed this otherwise.
A TLS middlebox in this approach can be a physical firewall device or a proxy, and additionally some antivirus and firewall programs, mostly on Windows, implement this functionality.
Note that cryptographic research has found that many TLS middleboxes contain security problems. For example, they may only support older, less secure algorithms or parameters; they may fail to validate certificates correctly or at all; they may not support the latest version of TLS; or they may fail to implement security-relevant extensions, like encrypted ClientHello. This is in addition to various cases where they are known not to implement HTTP properly and therefore break tools relying on it, like Git. Thus, unless you are sure your implementation doesn't suffer from any of these flaws, deploying such a middlebox is probably unwise.
It is possible to do some analysis on encrypted payloads. TLS supports padding, but it is less common with modern AEAD algorithms, so most connections leak some amount of information about the data being transferred. If the ClientHello is not encrypted, it's possible to receive the server name from the SNI extension, as well as information about the inner protocol via ALPN. These are used in some countries to implement censorship. Additionally, timing data is possible as well, but that is not always interesting. Provided you cannot decrypt the data, though, you cannot see what the actual content is.
