How do services with high uptime apply patches without rebooting?

Question

How are critical security updates installed on systems which you cannot afford to reboot but the update requires a reboot. For example, services/businesses that are required to run 24x7 with zero downtime, e.g. Amazon.com or Google.

Answer 1

There are various utilities in different operating systems which allow hot-patching of running code. An example of this would be kpatch and livepatch features of Linux which allow patching the running kernel without interrupting its operations. Its capabilities are limited and can only make trivial changes to the kernel, but this is often sufficient for mitigating a number of critical security issues until time can be found to do a proper fix. This kind of technique in general is called dynamic software updating.

I should point out though that the sites with virtually no downtime (high-availability) are not so reliable because of live-patching, but because of redundancy. Whenever one system goes down, there will be a number of backups in place that can immediately begin routing traffic or processing requests with no delay. There are a large number of different techniques to accomplish this. The level of redundancy provides significant uptime measured in nines. A three nine uptime is 99.9%. Four nine uptime is 99.99%, etc. The "holy grail" is five nines, or 99.999% uptime. Many of the services you listed have five nine availability due to their redundant backup systems spread throughout the world.

Answer 2

I watched a presentation at a security conference by a Netflix employee. They don't patch at all. Instead, when a patch is required, they stand up new instances and then blow away the unpatched ones. They are doing this almost constantly. They call it red-black deployment.

Answer 3

The short answer is:

They do reboot.

You seem to assume that Amazon and Google run on a single server, and if that is rebooted, the whole site/service is down. This is very far from the truth - large services typically run on many servers that work in parallel. For further reading, look at techniques like clustering, load balancing and failover.

Google, for example, has over a dozen data centers across the globe, and each holds a huge number of servers (estimates are 100,000-400,000 servers per center).

In such environments, updates (both feature and security updates) are typically installed as rolling deployments:

• pick some subset of servers
• install updates on the subset
• reboot the subset; in the meantime the other servers take over
• repeat with next subset :-)

There are other options, such as hot patching, but they are not used as frequently in my experience, at least not on typical large websites. See forest's answer for details.

Answer 4

You can check "Deployment Activities" under "Software Deployment". A common method is to use a Load Balancer in-front of your services and redirect traffic accordingly. In a technique called "blue-green deployment", you redirect traffic from "blue" to "green" servers. This does not have any user-side downtime, provided of course that the application can handle this properly, e.g. through stateless services.

Say your application runs v1 on the blue server and your load balancer directs traffic there. You can upgrade the green server (which does not receive any traffic) to v2. You then reconfigure the load balancer to direct the traffic to the green server. So, you have upgraded from v1 to v2 without downtime.

You can use the blue-green technique also as part of testing. For example, you configure the load balancer to direct 95% of traffic to the blue server (v1) and 5% to the green server (v2). This way you can test your new version, under less traffic and having less impact on users in case it has bugs.

Answer 5

Its pretty easy when things are clustered and proxied. Because you have many nodes capable of doing the same job (or several in the case of data repositories such as search engines, Hadoop file systems etc.)

Take a web search. You hit www.altavista.com. The DNS entry lists a half dozen IP addresses and your client hits one at random. Each IP is a Cisco router, which fans that trafic out to a random one of 8 physical front-end servers (48 total) on internal IP addresses. That server normalizes your query (removes whitespace etc.) then takes an MD5 hash of it. The MD5 decides which of 300 proxy servers that query goes to. That query is sent on to the proxy via a standard protocol like SOAP.

The front-end servers are interchangeable because they handle only transient demands of a single query. Outside worst case, a customer gets their query dropped. You use RRD data or other data collection to watchdog when a front-end server starts failing, and you reroute its traffic to a standby server. Same can be said of the Cisco routers.

---

The proxy first checks its cache. For a cache hit, it does localization blending and sends the answer back; done. If it's a "cache miss", the proxy fans out the query to the search clusters.

If a proxy goes down, again another physical machine can be swapped in for that proxy. It's a little more critical now, because the proxies are not interchangeable; each one "owns" a little slice of the search result spectrum. So if the 0x0000-0x00d9 machine goes down, the substitute must know to step in for that range. And worse, that substitute machine will have an empty cache, so every search query will be a cache miss. That will increase load on the search clusters proper by a tiny bit per downed proxy. That means if you bounce all the proxies at the same time, don't do it during peak search hours!

The search clusters have similar layering and redundancy, of course, and each segment of the search database resides on several nodes, so if a node goes down, other nodes can serve up that slice of the results.

---

I'm focusing on the proxy as an example. Communication into it is via SOAP, communication out of it is via some similar high-level protocol. Data in and out of it is transitory, except for the cache which is important to balancing search engine cluster load. The point is, that it can be swapped instantly at any moment, with the worst case result of a few searches timing out. That's something the front-end server would notice, and could simply send its query again, by which time the new proxy would be up.

So if you have 300 proxies, and it takes 1/2 hour for a proxy to recover its cache, and you can stand to have search engine load increase 20%, then you can swap 1 proxy every 30 seconds, so in any sliding 30-minute period, 60 proxies (20%) are rebuilding caches. Assuming there's even a pressing need to go that fast.

That example takes 2-1/2 hours to rollout, and if an emergent threat required a faster response, then you either endure the pain of more cache misses, or you down your service long enough to patch (but in the search engine example the cache misses will still be a problem when you come back up. I've watched the RRD graphs after an emergency DB reload and necessary cache flush, it is something to see.)

Of course usually the process can be patched, stopped and restarted without a full reboot. I have seen uptime of 2 years on production nodes.