15

Fastly WAF service was down a while ago, and it has affected a lot of major internet platforms and sites such as Amazon, PayPal, eBay, Spotify, HBO Max, the UK's main government website - Gov.uk, and many more. This has affected dozens of countries across North & South America, Europe, Asia, and South Africa.

How can one service on the Internet make such an impact? Isn't this a huge security risk?

schroeder
  • 123,438
  • 55
  • 284
  • 319
DxTx
  • 1,403
  • 2
  • 9
  • 20
  • 5
    All of those organizations are Fastly's customers and they chose to put Fastly's software between their own servers and their users. No more, no less. – hobbs Jun 09 '21 at 01:36
  • 6
    What do you mean by “ Isn't this a huge security risk?”? – Tim Jun 09 '21 at 14:27
  • 1
    +1 @Tim. To me, it's not a **security** but **availability** issue. It costs money to the companies relying on Fastly, but does not affect security – usr-local-ΕΨΗΕΛΩΝ Jun 09 '21 at 15:16
  • 4
    Along with integrity and confidentiality, availability is a [key aspect](https://en.wikipedia.org/wiki/Information_security#Key_concepts) of security. – 0x2b3bfa0 Jun 09 '21 at 16:10
  • Asking how a company has an impact isn't even about availability, it's something business related. Not even a remote connection to security. – nobody Jun 09 '21 at 19:12

4 Answers4

26

The internet is not as decentralized as you might think. A small number of companies that provide CDN and WAF services (e.g. Fastly, Amazon AWS, Cloudflare, Akamai) serve a disproportionate amount of the content that we consume on the internet. The sites that you mentioned in your question (and incidentally, StackOverflow and StackExchange as well) all relied on Fastly for content needed in order for these sites to function.

mti2935
  • 19,868
  • 2
  • 45
  • 64
  • 10
    Even "small" hosting companies can have a significant impact, as seen in this related not so new piece of news: https://www.bleepingcomputer.com/news/technology/ovh-data-center-burns-down-knocking-major-sites-offline/ – A. Hersean Jun 08 '21 at 13:18
  • 1
    @A.Hersean, Interesting read. Thanks for posting it. – mti2935 Jun 08 '21 at 13:20
  • 2
    That was not the best source for this news: the claim that OVH is the 3rd hosting provider is obviously false (for example, it does not appear in this top 10: https://www.hostingadvice.com/how-to/largest-web-hosting-companies/ ). – A. Hersean Jun 08 '21 at 13:30
  • 11
    @A.Hersean finding [the source](https://www.crn.com.au/news/worlds-third-largest-hosting-provider-ovh-opens-melbourne-office-461480) of that 3rd largest claim wasn't hard. 1) it's from 2017 2) it uses an entirely different metric than hostingadvice.com - number of websites hosted vs amount of physical servers, both having their merit. They are certainly one of, if not the, biggest players in Europe. – jaskij Jun 08 '21 at 23:52
16

It's all about getting Internet content to the end users (us) as quickly as possible. To make this happen firms like Fastly, Akamai, etc - called Content Delivery Networks (CDNs) - work with content providers (CNN.com, foxnews.com, stackexchange.com, etc) to ensure that a site's content can be delivered to the end user "faster" than if you relied on the "standard" internet routing protocols. This not free and companies have to pay to have their content, web pages loaded faster.

BTW: Internet routing was all about ensuring you got to where you needed to go - not about speed. CDNs have large regional or even global networks that are optimized for Speed and hook into the internet backbone. CDNs have even developed unique routing protocols (think BGP but, proprietary and faster) and some even have "edge nodes" which means they can cache content (think CNN's texts and images) and store it in a server that might be a few miles from your house. That way when you go to cnn.com and it loads quickly you are like "right on, this web site rocks". If it loads too slow - well you will go somewhere else. Companies like CNN understand this and will pay a CDN to get the content to you as quickly as possible so you won't leave.

MANY companies use CDNs to ensure that their content can be delivered to their audience quickly which means if the CDN has a major problem it's likely that many companies will be impacted - like with today's event. Having said that CDNs spend a lot of time, money and effort trying to have a distributed and fault tolerant environment so that one issue does not cause a large impact. In fact there is one CDN that has over 200K Edge nodes around the world in dozens of data centers and everything is redundant to mitigate the risk of a single or multiple issues impacting a large area.

CrunkAlpha
  • 161
  • 3
  • 32
    All that work to keep me from leaving the site, only to pop an overlay over the content while I'm trying to read it, achieving the same result! – Michael Jun 09 '21 at 01:48
  • 26
    Yes but that overlay popped up very quickly didn't it? :) – Rob Jun 09 '21 at 07:47
  • 3
    Using CNN as an example in an answer about CDNs makes this (otherwise great) answer a little hard to follow! – Tim Jun 09 '21 at 14:26
1

It's a popular service because what they have to offer is widely appealing. Widespread downtime is a risk with popular services. Of course you would also expect popular services to have less downtime than others in general, as this could affect their popularity. Popularity also tends to come with money, which usually means a bigger investment in resources that decreases downtime. But popularity also makes it a bigger target for malicious actors.

Companies aren't going to pick a less popular service just to avoid having downtime at the same time as everyone else in case that service has problems. That just doesn't make much business sense, as these outages tend to be quite rare and other services are probably worse, which would eliminate any advantage they might've gotten from not being down in those brief moments when everyone else is down. They're going to pick the service (or services) that best serves their needs.

In the context of the internet as a whole, you could perhaps argue having many other services (especially those that provide similar interchangeable offerings) depend on one service is a security risk in terms of availability. But it isn't really a security risk for any of those other services. How many other services depend on a service you're using doesn't really factor into your decisions when it comes to security (except indirectly, e.g. "many other services depend on it, so it probably doesn't have much downtime"). Of course if you rely on one service, that's a security risk (at least if it's not strictly necessary). In theory it would be better to add some redundancy by making use of multiple services that do the same thing, but that's a different question (it would be much more expensive, it may come with some technical challenges or it may not even really be possible).

NotThatGuy
  • 698
  • 5
  • 6
  • With regards to Fastly, it's a caching service, so there is a small security risk over accidentally caching per-user pages, and delivering them to other users, similar to Steam's 2015 security flaw. These are usually used by large companies that can be assumed to be doing this correctly, thankfully. – user3757614 Jun 09 '21 at 17:01
1

To pick up on this:

Isn't this a huge security risk?

The short answer is no, although it depends on what you mean by “security”.

It’s a risk to availability - as we have seen by the recent outage, but, as with many things, there has to be a cost-benefit analysis.

There are two alternatives to using a CDN service like Fastly:

  1. Host it yourself, with a large number of edge sites around the world, to ensure high speeds for every user, or
  2. Don’t use any CDN and accept that speeds will be slower.

Option 1 has an obvious cost: the purchase, maintenance, disposal and administration of hardware has a (fairly high) cost. There’s also still a risk here: why would you be able to do better at hosting a CDN than Fastly? Fastly is a company entirely focused on the provision of the CDN, and they employ industry leaders. Your company would not be! Option 2 is less obvious, but there will be users who don’t use your shop / read your blog / sign up for your service if they perceive your website to be slower than your competitors.

There are two costs to using Fastly: the service costs money, and sometimes the service fails.

The question (which is hard to answer) is: which costs more? Fastly or no Fastly.

Clearly, even large websites which do have significant presence in edge nodes around the world still see an advantage to using Fastly, notably Amazon - a subdivision of which (AWS) does offer a CDN! Now, perhaps that advantage has reduced, given how significant the recent outage was, but I imagine it’s still in favour of Fastly in a lot of use cases.

Tim
  • 950
  • 1
  • 7
  • 16
  • As a note there’s another reason to use Fastly. The saying goes “nobody got fired for buying CISCO”, and I think the same applies here! Sure, lots of big websites had downtime, but it wasn’t their fault. They get to say “sorry, Fastly screwed up”, instead of “sorry, we screwed up”. How valuable that is, I’m not sure, but the focus of the news headlines being “Fastly outage takes down major websites” faiths better (for them) than “Amazon.com suffers significant outage”! – Tim Jun 09 '21 at 16:19