125

I found out that my ISP does deep packet inspection. Can they see the contents of HTTPS connections? Wouldn't having HTTPS ensure that they can't see the contents being transferred?

And can having a VPN protect me against deep packet inspection by ISPs?

D.W.
  • 98,420
  • 30
  • 267
  • 572
cppanonhelp666
  • 1,233
  • 2
  • 8
  • 6
  • 9
    Immediately made me think of [this](http://4.bp.blogspot.com/-jZp1bLp3Q_A/TfhrQmNJvJI/AAAAAAAAAgI/wso4_M0pucs/s1600/Screen+shot+2011-06-15+at+10.19.45.png). – einpoklum Mar 28 '17 at 19:53
  • This is not worth a full answer, but I think it can be very helpful for understanding who can see what in your traffic. It's obviously designed for Tor, but there's also a HTTPS button: [Tor and HTTPS](https://www.eff.org/pages/tor-and-https). _data_ and _user/pw_ is what would correspond to "deep packet inspection". – pipe Mar 29 '17 at 18:36
  • 14
    how do you know your ISP is doing DPI? – Duck Mar 30 '17 at 05:42
  • 16
    @SpaceDog: it is required by law in some countries, which makes it very easy to tell that one's ISP is doing it! – dave Mar 30 '17 at 05:48
  • 3
    @SpaceDog, one of their employees anonymously accepted that they performed DPI to some news outlet. – cppanonhelp666 Mar 30 '17 at 16:56
  • 1
    Yes to VPN!!! All they would see is the IP of your VPN end point! If you are concerned at all and willing to spend money on VPN then do it, main reason I don't is the speed will be effected – FreeSoftwareServers Mar 30 '17 at 18:15
  • Is DPI even legal? In what country do you live? – Andrea Lazzarotto Apr 02 '17 at 13:26
  • @FreeSoftwareServers given that most VPN servers have connections whose bandwidth far exceeds the average home Internet connection in several countries, the slowdown would not be caused by the VPN (well OK, excluding the encryption overhead which might be considered negligible). – Andrea Lazzarotto Apr 02 '17 at 13:27

7 Answers7

140

Deep Packet Inspection, also known as complete packet inspection, simply means they are analyzing all of your traffic as opposed to just grabbing connection information such as what IP's you are connecting to, what port number, what protocol and possibly a few other details about the network connection.

This is normally discussed in contrast to the gathering of NetFlow information which mainly collects the information listed above.

Deep packet inspection gives your provider a lot of information about your connections and habits of Internet usage. In some cases, the full content of things like SMTP e-mails will be captured.

HTTPS does encrypt the connections but your browser has to make DNS requests which are sent primarily via UDP so that data will be collected as will any unencrypted links or unencrypted cookies sent incorrectly without https. These additional bits which will be collected may be very telling about what type of content you are looking at.

The larger concern for most people is about data aggregation, by collecting this information a data scientist could create a fingerprint for your Internet usage and later associate with past activities or activities from other locations (when you are at work or are on vacation). Likewise, your service provider may choose to sell this to any number of organizations (possibly including criminal organizations) where it could then be used against you in ways. In many countries, people have an expectation that their communications are considered to be private and collecting this data very much goes against that privacy expectation.

Another interesting aspect of this is in the cases like the US where this data may soon be sold it allows International communications sent to people, or servers, in the US be sold as well. Likewise, this could potentially allow every agency from local law-enforcement, military, tax authorities, immigration authorities, politicians, etc. a way to bypass long-standing laws which have prevented them from accessing this type of information, or important informational subsets within this data otherwise.

A slightly different concern when this data can be sold is competitive intelligence / corporate espionage. In the scenario where a company does a lot of research-intensive work at their headquarters located in some small geographical location (think of pharmaceuticals or a defense contractor) selling that data makes it possible for anyone to buy all of the traffic from the local ISP where most of those researchers live and analyze what they are looking for when at home, possibly even directly from the ISP hosting the traffic for their corporate headquarters. If other countries aren't selling similar data it gives foreign companies and companies wise enough to try and buy this data a huge technical advantage. Likewise, it would also allow foreign governments to buy ISP traffic which includes the data from US (or other government) Officials homes.

Imagine companies monitoring their employee's behavior at home or on their mobile devices.

This will likely have a chilling effect on activists and whistle-blowers as well.

Likewise, if credit cards or PII are sent in the clear to a poorly secured remote site your ISP's data set now has a potential PCI or PII regulatory issue on their hands. So this amplifies data-leakage problems of all types by making additional copies of the data leaked.

With the examples I've just mentioned above, and there are hundreds of others, it should be easy to see why this type of data collection has a different level of importance to it than just metadata or basic connection information. Even if your ISP never sells this data they are collecting quite an interesting dataset.

It's a security issue that definitely has a lot of potential long-term security implications.

Trey Blalock
  • 14,099
  • 6
  • 43
  • 49
  • 52
    "In many countries, people have an expectation that their communications are considered to be private and collecting this data very much goes against that privacy expectation." It seems certain members of the US government aren't fond of that, though. – JAB Mar 27 '17 at 23:42
  • 47
    In many EU countries the sale of this data would be against people's privacy expectations AND against the law. Other countries aren't as concerned about privacy. – dave Mar 28 '17 at 02:08
  • 47
    At least in germany, deep packet inspection is illegal. – Magisch Mar 28 '17 at 06:40
  • In France, DPI is common place, at least for one major historical ISP (I let you guess which). – A. Hersean Mar 28 '17 at 08:59
  • 1
    If you're doing DPI there's a fair chance you can exploit imperfectly implemented security to see inside HTTPS etc. as well. Certainly the security services can. My basic assumption is that *someone somewhere* can see everything I do online. – John U Mar 28 '17 at 10:49
  • 1
    @A.Hersean: I find it all the more surprising seeing how heavily regulated the interception of telephone communications are (or at least was when I did my internship... 10 years ago). – Matthieu M. Mar 28 '17 at 11:47
  • 11
    Even without the DNS aspect, TLS with SNI (which is what is commonly used today) transmits the hostname in the clear in the initial TLS exchange, before encryption is set up. – user Mar 28 '17 at 12:29
  • 1
    @A.Hersean I'm pretty sure they are unable to sell that information to third parties though... so the only thing they could do is like: guess how much traffic you use in streaming vs other stuff and bill the amounts differently or something like that. – Bakuriu Mar 28 '17 at 17:10
  • 1
    @Bakuriu FYI: There is a Bill going to Congress right now to allow ISP's to sell this data. https://www.eff.org/deeplinks/2017/03/five-creepy-things-your-isp-could-do-if-congress-repeals-fccs-privacy-protections – Trey Blalock Mar 28 '17 at 17:15
  • 2
    @TreyBlalock That's irrelevant... at least until the USA invade France. – Bakuriu Mar 28 '17 at 17:37
  • 9
    @JohnU: That's unlikely and your claim is a severe disservice. There is no way to "see inside" HTTPS without active attacks that not only inspect the content but change it. Modern client software will not allow this unless it's been backdoored (possible if ISP tricked you into installing software they provided), and savvy users **will notice** any such active attacks, making them high-risk for the attacker. So for practical purposes, no, nobody is "seeing inside" your HTTPS. – R.. GitHub STOP HELPING ICE Mar 29 '17 at 18:11
  • 1
    @Magisch: would you have any sources for that claim about illegality of DPI in Germany? This would severely impact a lot of "intelligent" firewalls which rely on some specific communication patterns to identify the traffic. – WoJ Mar 29 '17 at 19:22
  • @WoJ I think it's illegal to do for ISPs only. I'll look into it further and dig up the source when I get time. – Magisch Mar 29 '17 at 20:03
  • The DNS information leak can be mitigated with DNS/TLS (which is not widely supported). But then SNI, leaks the target domain anyway. – ysdx Mar 30 '17 at 10:17
  • @R.. From what I've read from the likes of Risks Digest, Bruce Schneier, etc. it's by no means impossible/implausible for there to be bugs, flaws, misconfigurations etc. even disregarding backdoors or other covert access going on. Whatever the odds, since they're less than infinite it's good practice to assume that someone who wants to see your communications _could_ do so, and act accordingly. Legality is a moot point, we don't expect our spooks to play gently when they're protecting our security and we certainly wouldn't expect gentlemanly conduct from bad actors. – John U Mar 30 '17 at 15:26
  • 6
    Just to be pedantic: *"in the cases like the US where this data can now be sold"* Presuming you are referring to recent rulings, the phrasing "can now be sold" is not quite accurate, as there was never a point in time when it *couldn't* be sold -- those privacy protection acts that were canned weren't set to take effect until the end of this year anyways, so nothing actually *changed*. The ruling stopped the change that was *about* to happen soon. There is no *new* risk, however. I thought it was important to point this out as there has been a lot of misconception on this hot recent news item. – Jason C Mar 30 '17 at 15:26
  • 2
    @JohnU: They're all active attacks, which mean they can't be deployed at scale/dragnet because the probability of being caught rapidly approaches 1.0 as the number of victims increases. Possibility of a passive attack is essentially equivalent to breaking the underlying cryptography. – R.. GitHub STOP HELPING ICE Mar 30 '17 at 17:33
  • 2
    Also, some comments mention SNI but even without it the server would present it's certificate unencrypted which contains CN usually. – Cthulhu Mar 30 '17 at 19:12
65

Trey Blalock's answer describes percisely what deep packet inspection (DPI) is. But I'd like to add three things to hopefully answer your specific questions:

  1. There is a technique of DPI that does decrypt your data, called SSL interception, although it is more common in enterprise situations and only possible if the ISP (or any other interceptor) has the ability to install a certificate on your machine. So unless the ISP had some way of doing that (technician etc.) this is probably off the table.
  2. HTTPS would prevent the ISP from being able to read data. Of course, This is only true for services that use HTTPS (which unfortunately is not all of them). Also you need to consider that the ISP can read metadata whether the connection is encrypted or not.
  3. A VPN would protect you against DPI performed by the ISP (not by the VPN provider). This is thanks to the fact that VPNs use an encrypted tunnel to connect you to the 'exit node'. This encrypts all of your traffic, and all of the metadata will show packets leaving your computer and going to the VPN server (thus not disclosing the actual server you are accessing).
Community
  • 1
MiaoHatola
  • 2,284
  • 1
  • 14
  • 22
  • 55
    Note that some ISPs (for example Deutsche Telekom) are accepted Certificate Authorities themselves. – Jonas Schäfer Mar 28 '17 at 06:58
  • 4
    @JonasWielicki Great point. That must be taken into consideration! – MiaoHatola Mar 28 '17 at 07:13
  • 1
    You could still have a DNS leak using a VPN which can let your ISP know which domains that you are visiting over it. – Emil Hemdal Mar 28 '17 at 08:16
  • @JonasWielicki although this would be illegal for them to do because of EU laws I think? – Tim Mar 28 '17 at 08:36
  • @Tim I don’t know about EU (or German) laws regarding this. But considering that we have a state-supported "secure" E-Mail service where the MSPs are allowed to decrypt supposedly "end-to-end" (no, it’s not, unless you count the MSP as "end") email to perform spam- and virus-checking, I’m not sure I should trust in those laws. – Jonas Schäfer Mar 28 '17 at 10:13
  • 1
    Using HTTPS only encrypts the contents of a page right? So regardless of DPI wouldn't my ISP know that I am visiting this particular website? – cppanonhelp666 Mar 28 '17 at 13:08
  • 1
    @cppanonhelp666 Yes. I've noted that at the end of the second point. – MiaoHatola Mar 28 '17 at 13:13
  • 8
    @JonasWielicki While true, it wouldn't be for long if they used their CA status to do DPI. Between WoSign and Symantec (although to a lesser extent), I don't think they'd be spared if they generated certificates to intercept SSL connections. – Ginnungagap Mar 28 '17 at 16:15
  • @Ginnungagap Right, the monitoring of certificates CAs got better recently. – Jonas Schäfer Mar 28 '17 at 16:18
  • 2
    @JonasWielicki: While there is a theoretical/technical threat of such abuse, it would be illegal in many jurisdictions and would be a violation of the policies CAs are required to follow in order to remain trusted by the browser. At any nontrivial scale (basically, any untargeted attacks or targeted attacks of savvy users) this would be quickly caught. – R.. GitHub STOP HELPING ICE Mar 29 '17 at 18:19
  • 1
    @emilhem I would expect all DNS traffic to travel (securely) over the VPN connection. Is this not the case? At any rate, the VPN has to come out somewhere, and must connect to the network through a (possibly different) ISP, so you're really just pushing the problem somewhere else at that point. – jpaugh Mar 29 '17 at 19:37
  • Point 1 can be mitigated/unmasked using HSTS though that has to be implemented by both the browser maker and the site, though that wouldn't stop the initial headers from being visible. – James Snell Mar 30 '17 at 10:28
  • 1
    RE Point 1, I know I've seen at least one question either here or on superuser, from a person whose ISP was forcing them to manually install an MITM certificate (IIRC to support govt snooping) in the last year or two. – Dan Is Fiddling By Firelight Mar 30 '17 at 14:18
  • @jpaugh not always. It depends on what kind of VPN software and its version. Making a simple search for "DNS leak [VPN software]" displays the issues in the different VPN softwares and how to mitigate against DNS leaks. – Emil Hemdal Mar 30 '17 at 17:02
  • @emilhem Would maintaining a local DNS server mitigate this, or does that open up a different set of problems? – Kenneth K. Mar 30 '17 at 19:19
  • @KennethK. maybe. It depends on if your local DNS server uses the VPN or not. In fact it could be worse it it doesn't use the VPN since the actual DNS lookup comes from your network and not from a public DNS server (like Google DNS) but you ISP could use transparent DNS lookups so you might be screwed anyway. I use https://dnsleaktest.com/ to check whether or not I have a leak. I also use https://www.dnscrypt.org/ – Emil Hemdal Mar 31 '17 at 14:22
  • @Magisch I suppose that (as always in legal matters) the answer is "it depends". In the extreme case, deep packet inspection (and manipulation) may even be legally *enforcable* in order to install e.g. a Bundestrojaner ("Federal Trojan"). In other cases, deep packet inspection (and manipulation) with the purpose of removing malware *may* be legal – Hagen von Eitzen Apr 01 '17 at 16:05
  • 1
    Some of us have been waiting/expecting/fearing years for ISPs to contractually require customers to place an ISP root certificate in the customer's root certificate store. Or, equivalently, for the UK to legally require it. That would allow the ISP to proxy and inspect all traffic. Apps like Google Chrome would detect such man-in-the-middle interception, but cannot prevent it. // Wifi on US domestic airplanes usually does this, but often allows Google search traffic through unmolested, to reduce complaints. // VPN and TOR won't help, unless steganographic. – Krazy Glew Apr 02 '17 at 05:29
41

As stated by Trey, DPI can see the entire content of your network traffic. All of it. If it is plain text, then they see everything that you do.

To add on to Miao's answer:

Things DPI can see, even when you use HTTPS:

  • DNS information, e.g. https://catvideos.com/tigers - they will see https://catvideos.com
  • IP address connectivity. So, even you you HTTPS to that site with cat videos, they can see that you connected to that cat video site and downloaded 500 GB of data. They don't know what data, but they know the DNS name, and the IP address, and amount of data to that site, and to every site.
  • Ads. Many/most Ad networks do not use HTTPS, so that data is not always encrypted. This can result in a "mixed encryption" or similar warning from a browser.
  • other data: Many sites using HTTPS for login will then drop the encryption for everything else.
  • graphics: many sites won't encrypt things like their logo or various graphic or video files. They may encrypt your login and search, but not results.
  • other non-HTTPS traffic like UDP, mail, SNMP, ftp, telnet, updates to some software might not use HTTPS, etc.

With a VPN, they will still see 100% of the data. However, other than the connection to the VPN provider, they'll only see encrypted data. They will know that you downloaded 800GB from VPNco.com, but will know nothing of the data inside. Even things that are not encrypted via protocol will get encrypted since a lower level is encrypting. Now, the VPNco.com will then see your data.

With the (potential) change in US law about ISP and data privacy, combined with the (potential) loss of net neutrality, ISP's might be able to not only see 100% of your data, they could modify that data, slow or block sites they want, and might be able to sell any/all of your data to a 3rd party (as Trey states).

I'm not covering MITM (like Miao states in #1 above), since you stated ISP, I assume that you are talking about a home system and a DSL or cable modem.

https://stackoverflow.com/questions/499591/are-https-urls-encrypted

MikeP
  • 1,159
  • 7
  • 12
  • 10
    So ultimately it boils down to whom does a user trust​, ISP or the VPN provider? For VPNs based in, say USA when they state that they do not keep logs (like PIA) won't NSA (or CIA) force them to either keep logs or close shop? – cppanonhelp666 Mar 28 '17 at 13:14
  • 9
    @cppanonhelp666 Don't trust VPNs based in the US, because the government will do their best to extract that data. –  Mar 28 '17 at 20:45
  • 2
    You can replace "(potential) change" with "change" now. – JAB Mar 30 '17 at 14:55
  • 1
    So, who do you (dis-)trust more? ISP, VPN, CIA/NSA, etc? I feel that at least with CIA/NSA, they are looking for national security issues not watching Netflix from a different region or downloading cat videos. The ISP and whomever they sell to might be interested in that data. YMMV. – MikeP Mar 30 '17 at 19:35
1

With deep packet inspection, the ISP can detect most VPN protocols (not the data encrypted in the VPN packets, just that there is VPN traffic) and block it. Some companies do this to ensure that they can decrypt all traffic (with the MITM attack and forged certificates to have DPI on SSL as well). The idea is to force you to use "insecure" communication channels by preventing everything else. Note that these "insecure" channels might be more secure , from the company's point if view, as they can do Data Leakage Prevention there.

In such a case, non-standard VPN techniques, like HTTP tunneling might be an option.

Note that the terms of use might disallow measures to circumvent DPI.

Edit: some ISPs use DPI for traffic shaping. They don't log all the transmitted data, they just check (for example) for BitTorrent traffic and assign it a lower priority or limited bandwidth. Now, they are not stealing your password, just the bandwidth you are paying for....

Klaws
  • 149
  • 3
0

If you don't trust your ISP then your first priority should not be about packet inspection at all, but rather be to establish a trusted second channel of communication for which you can exchange information about circumventing such things.

As long as you rely solely on your ISP as the lone channel for all information exchange they can technically send you wrong login info to your VPN, even if they don't they could still take over any encryption handshakes attempted because they will always be in the middle.

They could have people in their employ who are bribed to do so or be required by law for any reason.

mathreadler
  • 109
  • 4
-1

All the above is true. One more thought: Did your ISP give you a self-signed root certificate and is it in your browser? If they did, they can open your HTTPS traffic.

happyTroll
  • 1
  • 2
    This has already been covered in the first bullet point of [MiaoHatola's answer](https://security.stackexchange.com/a/155062/16960). – Xiong Chiamiov Apr 02 '17 at 19:04
-2

You have installed a vendor security certificate on your computer, otherwise there was no such option, and now there is a question how to bypass this test

usrbs
  • 1