Disclaimer: I'm going for a relatively high-level understanding. If you want a detailed guide, that's out of scope. Additionally, there are other ways (entirely in software) to implement virtual machines to which this does not apply. I am also focusing on "breaking out" through the virtualisation mechanisms only - i.e. not ones that can happen PC-to-PC on actual hard networked hosts.
I like detail, so here we go with some. Firstly, codeproject has some excellent assembler references on the different modes of an x86 CPU (real, protected and long) and use of virtualisation. There's an Intel VT blog (I'm not sure if Intel write this) and lastly the first part of the Rootkit Arsenal is dedicated to explaining x86 and is an excellent read, complete with walkthroughs and nice diagrams. Understanding it all takes patience, so what I'll do here is to give a very brief introduction to how it works.
The way we rocked when we ran DOS
DOS and early 16-bit real mode systems operate a segmented memory model. There is no control over the size of segments and there are no protection switches on any of those segments. Code gets loaded into a segment of memory and it runs; it can far jump into other segments, so any code, anywhere can alter anything, including producing a TSR (terminate and stay resident) piece of code that simply points one of the IVT entries (interrupt vector table) at an address in its space, before executing the original. Basically, there is no protection. None. Nada.
The rise of 32-bit protected mode
Protected mode gets complicated quickly. There are three parts to it - segmentation, paging and PAE. Each requires a table of data that tells the CPU about that segment, page or helps it extend the address space (PAE). These include the famous ring flags (they apply to segments and pages) which implement process isolation. Paging is your way to load data out of RAM and onto disk and create fancy things like virtual memory (see, the word virtual! We're getting there!)
Long mode
Long mode does away with segmentation and simply mandates the PAE/Paging structures. Again, to totally trivialise implementing an OS, Paging is controlled by structures in memory which are then set up via special instructions. Voila, one can achieve process isolation with the right settings. Again, I'm trivialising slightly...
Give me virtualisation!
Okay. Virtualisation is the same general concept. Virtual machines are set up using virtual machine control structures which dictate how their memory is mapped back to physical memory, kinda like paging. Crucially, under certain conditions the virtual machine will be required to ask the host operating system for something, kinda like process isolation, kinda like a software interrupt. These are referred to VM exits and provide information to the host such as the state of registers on exit. Kinda just like a system call.
Can a piece of malware break out of a virtual machine?
So, as far as the VM is concerned, the host OS has all its own memory space and can be infected/damaged/destroyed as it pleases.
In terms of affecting the host memory directly, the virtual machine cannot, because it cannot see it. The host must map the required memory into the virtual machine space. It must also, in that memory space, implement everything from the BIOS up. In order to communicate with certain host devices for certain tasks, the host machine must set up those VM exit conditions and the target VM must trigger them. When that happens, control is transferred to the host.
There are, therefore, two possible at-risk areas:
- The actions the host takes in response to a VM exit. If there are any bugs in this handling it may be possible to persuade the host to execute something it shouldn't.
- Any host access to the guest machine's memory space. Remember the host machine code running in ring 0 can waltz in and crash the party wherever it so pleases. It just so happens that you can set the guest's memory from the guest (surprisingly).
This leads you to your exploit mechanism. You need a handling bug in the VM exit routine, then you need to be able to persuade that code to execute some memory, ideally code you just put into a page from the guest vm. Once done, say bye to Kansas.
As Tom Leek says, VMs are incredibly effective in defending against fork bombs. In a similar vein to the way the OS can limit how much memory a process can allocate, so it can limit how much memory is mapped to the VM. Run out and the guest OS believes it is out of physical memory; the host will not allocate it more unless you implement a VM exit to do this, which would be a bit dangerous and I do not believe this is done.
How likely is this?
Not very. It depends on those VM exit implementations entirely, or reading memory from the guest on the host with a nice bug in your reading code. It also requires that said bug would allow you to control the crash in such a way that you can force execution to the memory address your host holds. The VM exit must be able to access that memory.
What I have not covered?
- Attacks on existing software stacks such as TCPIP. The vulnerabilities here are the same as if you had two actual phsyical PCs anyway.
- Entirely software implemented virtualisation.
- Virtualisation on any other type of chip. This applies to Intel VT compatible setups.
Finally, I have previously argued that process isolation is a form of sandboxing. Reading that answer and this one, you should now be able to understand why I define them that way. There are remarkable similarities between process isolation and virtual machines in x86.
Update
So, I've dug around even more on this, especially into the blue pill research. What I've described is a very simplistic high level view. I've found more detail. Here's a whole paper dedicated to it from Invisible Things Lab. It turns out that their defences talk included the concept of denying execute access to user mode pages from ring 0, thereby preventing direct execution of data the virtual machine has placed into memory. It turns out this is being implemented in Intel CPUs and patches currently exist in the Linux Kernel. So, depending on how that goes, it may well be the case that attacks of this nature become much harder, even if exploits exist.