Did US and UK spy agencies defeat privacy and security on the internet?

Question

This question is meant as a canonical question in regard to the US and UK spy agencies compromising end nodes and encryption between nodes to spy on people they suspect to be terrorists. However, this has the side effect of significantly elevating the risk of exposing innocent people's personal data.

Recently an article was published by the Guardian detailing:

US and British intelligence agencies have successfully cracked much of the online encryption relied upon by hundreds of millions of people to protect the privacy of their personal data, online transactions and emails, according to top-secret documents revealed by former contractor Edward Snowden.

What is the impact of this and have they really broken all of the crypto out there?

Answer 1

There will be a lot of speculation regarding this question. I will try to provide as much information as stated in the articles. I will also update the answer regularly with facts provided in the comments.

Relevant articles to this answer:

• NSA: How to remain secure against surveilance
• The US government has betrayed the internet. We need to take it back
• US and UK spy agencies defeat privacy and security on the internet

First of all I would like to say:

Is this threat real?

Depending on how well we can trust the papers, it should be considered a real threat in the sense that security agencies have succesfully implemented backdoors in software or at encryption end-points. I believe this is probably true as three reputable news papers were kindly requested to not publish the article. This means that there is a high likelihood that at least part of the story is true.

Have they really broken crypto?

As far as we can tell from the articles they mainly have three strategies:

1. Use supercomputers (clusters) to brute force encryption protocols. This probably means they can efficiently bruteforce encrypted files.
2. Implement backdoors into the software which does the encryption.
3. Make technology companies comply with their demands, some of which may include #2.

Option 2 and 3 suggest that they have not succeeded at real time decryption of, for instance, SSL. As Bruce Schneier stated:

The NSA deals with any encrypted data it encounters more by subverting the underlying cryptography than by leveraging any secret mathematical breakthroughs. First, there's a lot of bad cryptography out there. If it finds an internet connection protected by MS-CHAP, for example, that's easy to break and recover the key. It exploits poorly chosen user passwords, using the same dictionary attacks hackers use in the unclassified world.

They still require people at either end node to implement a backdoor covertly or make the technology company help them in decrypting traffic passed through their systems. Chances are high they have the encryption/decryption/signing keys of some of the Certificate Authorities, which would allow them to setup proxies and perform man-in-the-middle attacks. Due to the trusted certificates (because they can sign them themselves), these attacks will not be noticed by the users they spy on.

Note that they have proposed a system to perform real time decryption but there are not indicators that they have actually succeeded in building this. All we can do, at this moment, is speculate.

There has also been one case where they, allegedly, backdoored a random generator (according to Wired) used by encryption algorithms.

Please note: Encryption standards are public which means anyone who wants to scrutinize them can look into them. The NSA has made it a lot more difficult to review them though. (Implementations are a whole other thing though.)

Note that for option 2 it seems they specifically target commercial software. If you want to be more confident you're not using compromised software, you should use open-source products and compile the binaries yourself. (Although, theoretically speaking, the compiler could also be backdoored.). The code could be peer reviewed, or you could review it yourself. (Unfortunately, the latter is often not feasible or practical.) Furthermore, this quote from Edward Snowden would also suggest that they haven't managed to crack strong crypto:

"Encryption works. Properly implemented strong crypto systems are one of the few things that you can rely on," he said before warning that NSA can frequently find ways around it as a result of weak security on the computers at either end of the communication

What are the consequences?

1. The NSA endangers everyone on a certain system by deliberately installing backdoors. They are not just risking the privacy of those they are investigating, but everyone using the system.
2. Open source software is less likely to be compromised. It's clearly stated they attack commercial, closed source software. Backdoors are much more likely to be spotted, and spotted more quickly, in open source software.
3. Commercial software, especially those published in the U.S., are more likely to be backdoored than any other software. (This is personal speculation, but I believe that it would be harder to do in many other regions - e.g.: the E.U. - due to more stringent privacy laws and the fact that multiple governments would often need to be involved. However, this is still no guarantee.)

It also seems that they want to compromise as many internet nodes/hubs (tier 1 providers probably) as possible. This is logical because most traffic on the internet will pass by the tier 1 providers at some point.

The biggest risk is data leaking because of their negligent practices with introducing backdoors. If the NSA is really after you I doubt some crypto will help you to save your ass. They will probably just round you up at some point and make you disappear. The NSA is not generally going to come after copyright infringers or script kiddies/hackers. They're more interested in the hard-core, dedicated (cyber) terrorists. I highly doubt that, unless the have indicators that you are a terrorist, they will use their information to sue you or even pass that information to another agency.

The danger within the NSA is however when someone like Snowden, but who has bad intentions, decides to leak all your private data or use it for personal gain (or any other purpose that is not in the interest of the citizens the NSA tries to "protect"). They have very limited oversight - and much less so publicly - at the moment, which greatly serves to facilitate abuse of the system.

What can I do?

Start by reading the article NSA: How to remain secure against surveilance written by Bruce Schneier.

My personal opinion is that the NSA probably has access to tons of sensitive data and that, even when using strong crypto, they still will be able to get access to sensitive data due to backdoors they introduced in systems or because of the cooperation companies give the NSA. There are some precautions you can take:

• Use strong passwords
• Use strong cryptography (websites with SSL certificates should be verified to be running a secure, strong version of TLS)
• Use VPN/proxies/Tor (not located in the US or UK - maybe not even Europe - though even they can still be backdoored)

We also need to open up software and protocols, as Bruce Schneier said:

We can make surveillance expensive again. In particular, we need open protocols, open implementations, open systems – these will be harder for the NSA to subvert.

My 2 cents

It's also an illusion to think the NSA are the only ones doing something like this. It would surprise me if the Chinese and the Russians (or any other state with a large secret police budget for that matter) didn't have similar programs. For the Chinese we already have indicators (APT-1) that they are involved in similar practices as the NSA. Does this make it any less wrong/hypocritical of the US/UK? Probably not. As Bruce Schneier said:

I am saddened to say it, but the US has proved to be an unethical steward of the internet. The UK is no better. The NSA's actions are legitimizing the internet abuses by China, Russia, Iran and others. We need to figure out new means of internet governance, ones that makes it harder for powerful tech countries to monitor everything. For example, we need to demand transparency, oversight, and accountability from our governments and corporations.

Answer 2

What is the impact of this and have they really broken all of the crypto out there?

The real impact is that any individual persons' private information is much less likely to remain private. The actual risk depends on each individual and their national connections. Clearly Iranian nationals with attachments to the US have a much higher risk than UK nationals with attachments to Antartica.

In my opinion it is impossible for any singular national agency to have broken all cryptographic protection.

Recall than an individual implementation of an algorithm is not the same as an algorithm itself. For example you might break the crypto-js - JavaScript implementation of Triple DES, but not necessarily the Microsoft .NET implementation of Triple DES.

Even more complicated is the use of an implementation by software for a particular purpose. An ATM machine may use a IBM implementation of Triple DES for authentication while relying an IBM implementation of Single DES for confidentiality.

If I was picking targets I would look for specific commonly used software with vulnerable cryptographic implementations. It is far easier to compromise software's effective use of cryptography then the cryptography itself.

Answer 3

As much as I'm interested, I'm afraid this question invites speculation, not known facts. So I'll add my own speculative idea to the list.

They could have copies of the private signing keys of some of the CA root authorities. That would let them create false certificates to enable transparent MiTM proxies (similar to BlueCoat, but without the warnings.) Such active interference might not go unnoticed if it were performed on a widespread basis, but it could be used for a few specific intercepts on an unwary target.