How do big cloud providers guard against VM escape attacks?

Question

I think it's pretty much unassailable to say that cloud computing as we know it depends on the concept of the robustness of virtual machines. Where one can depend on the security of VMs they allow workloads from multiple customers of a cloud provider to securely run on shared hardware, while also guaranteeing that if the security of something inside one customer's VM is compromised the host OS & firmware of the provider's server won't be affected. If you're Amazon, Microsoft, Google, IBM, or really any cloud provider of any size making sure VM isolation of each customer's stuff remains uncompromised is vital to your operations and to customers continuing to trust your service offerings.

However, as we know VMs are not escape-proof. Security vulnerabilities have been periodically found in every major virtual machine technology that I can think of.VENOM for Xen VMs is probably the most famous of these, but Hyper V, VMware products, and so on have also been afflicted by vulnerabilities. Of course, these known flaws can and are patched by providers. But there undoubtedly other flaws out there, waiting to be discovered and (potentially) exploited by attackers. In fact, one would assume that well-resourced attackers--meaning ones capable of developing new vulnerabilities & exploits in-house or buying valuable ones grey/dark markets--would have very strong incentive to devoting good amounts of time/effort/money to finding or buying novel ones that allow breaking out of the VMs used by major cloud providers. Especially given the ever increasing role that cloud computing plays in the tech infrastructure of so many companies & organizations that are, in turn, all potentially juicy targets (for varying reasons) to sophisticated attackers.

So, how do Amazon, Microsoft, etc. guard against the threat that a sophisticated attacker (whether state-sponsored APTs, high-end organized crime, or whatever) can develop for buy novel VM escape exploits and use them against the provider's infrastructure and customers? The one obvious measure that sticks out to me is running customized host-based intrusion detection systems on all cloud servers to try to catch VM escape events as/when they happen, but as we all know by now HIDS are hardly fool-proof. What other measures could/do cloud providers take to detect, prevent, or mitigate the danger of VM containment breaches?

Answer 1

Highly customized and patched hypervisors, sandboxes around said hypervisors to mitigate breakouts, and heavy monitoring. Of course, any given server only hosts so many VMs, so a breakout is fundamentally limited to a finite number of guests, if it's able to get past the protections outside the hypervisor. For example, QEMU can be compiled with a hardened toolchain, isolated using chroots, mode 2 seccomp, mandatory access controls. It can be patched to be made more secure, using custom seccomp filters, disabling excess code or features, adding custom, hardened drivers. Custom kernels can be configured and hardened to limit the ability of an escaped guest to compromise the kernel and escape mandatory access controls.

Unfortunately, the answer to the question can partially be answered with "they don't". Many cloud providers do regularly get compromised, they just don't know it, or don't disclose it. I see this happening periodically, and it's not pretty. Sometimes 0days are used, and sometimes little-known public issues with the providers are leveraged. For example, Amazon's higher end dedis not shutting down when switching customers, resulting in their GPUs not wiping VRAM, letting the next customer see what the previous one was doing (stealing secrets in OpenCL, anything which was sent over VNC or with X11 forwarding, etc). The Amazon security team knows about this, but still have not fixed it. They do not tell their customers this. This shows a lack of effort put into security, and a lack of awareness about extant security issues. With something like this which you'd think would become big news being known by few people, it's not surprising that large-scale or advanced attacks against cloud providers can happen without the general public being aware of it. The lack of awareness and news of attacks leads people to wonder how the hell these large businesses stay secure. They aren't.

To my left is a window with vim open. In it is code with a 0day on used recently to break out of a hypervisor on a certain medium-sized cloud provider. No protections were in place. All that happened was the exploit broke out and gained access to the host with the privileges the process the hypervisor was running as. No sandbox or extensive monitoring solutions were in place. A single obsolete syscall causing a kernel vulnerability lead to a compromised kernel and persistence in memory (grsecurity would have mitigated it, fwiw), and subsequent exfiltration of information. This company is worth hundreds of millions of dollars. I'm sure countless people have been in there before. Although this is not real evidence and just my own experience, it hopefully should show that cloud big cloud providers do not in fact guard against VM escapes well at all, with the exceptions mentioned in the first paragraph.

Answer 2

The primary defense against vm escape attacks specifically is defense-in-depth. Specifically that if there are varied and plentiful defensive mechanisms in place that an attacker will trigger enough of them that the defender can stop the bad things from happening. Even if an attacker successfully breaks out of one vm in one hyper-visor, there can still be things that catches them (Tripwire, anti-virus, Modsecurity, for examples). Create enough layers that do not share vulnerabilities and it become extremely hard if not impossible to break out.

In a lot of ways though, your question hits the nail on the head. A skilled and motivated attacker, capable of building an attack unique to all the defenses, could implant a hyper-visor rootkit and the host would never know.

A good team of defenders should be able to understand and use a wide variety of tools/tricks to shut down attackers. Hopefully this is enough to get you started the right direction.