How does advanced firewall scan data (check for signature etc) if it is encrypted?

Question

Let's say for normal web browsing, since it's using HTTPS, all the data will be encrypted. In that case, how does the firewall/IDS/IPS check for signatures to detect anomaly/virus/malware?

Answer 1

For actually applying signatures to the decrypted traffic the firewall needs to have access to this traffic. The typical way to do this is by essentially doing a man in the middle attack, only that the "attacker" (the firewall) is considered trusted in this case. See Is it common practice for companies to MITM HTTPS traffic? for more.

Another way sometimes employed in firewalls/IDS protecting a specific server (like in web application firewalls) is to not make this server the TLS endpoint. Instead the firewall itself or some device in front of it is the TLS endpoint so that the firewall can see the plain traffic. A variation in this scenario is to provide the firewall the the private key of the servers certificate so that it can decrypt the traffic. But this works only if RSA key exchange is used while today DH key exchange is recommended.

And then there are attacks which can be detected without decrypting the traffic. These signatures for example rely on specifics of the TLS handshake like the kind and the order of ciphers and TLS extensions used or information about the certificate send by the server. Since the certificate is send in plain text in the TLS handshake one can determine the CA which signed it or if it was self-signed and similar by simply passively sniffing the traffic. See Deciphering Malware’s use of TLS (without Decryption) for a nice paper about how one can detect malware communication this way.