Looking for tips: building a workstation around Xen and GPU pass-through

Question

I'm building my new home machine right now and as usual this will be a heavily overloaded workstation that will serve as development and testing machine, file server and game console (plus I'm adding TV cards and multimedia streaming for my home network).

Since these use cases work against each other (file server is unavailable when the machine is used as game console), I'm heavily considering the use of Xen to provide everything at once. Performance shouldn't be an issue in general, but the game console use case definitely requires direct access to a GPU.

Therefore I have several questions:

• is this feasible in general?
• is the GPU passthrough usable enough with non-server GPUs?
• when buying, what technologies should I look for on the motherboard (most likely choosing Gigabyte + Intel CPU again)
• is there some virtualization technology creeping on the horizon that is worth waiting for?
• how good is Xen at dividing resources?
  • the file-server and multimedia-streaming portion of the workstation has to work even if the game console part is eating 100% of its resources and vice-versa
• the system will run on top of 8 disks with software Raid6 and LVM, should I buy a dedicated disk for the game console portion of the system?
  • I have read about some problem with Xen and I/O performance

Any input will be greatly appreciated.

P.S. I won't mind using other technology then Xen, but I'm unaware of any other virtualization platform that supports GPU passthrough.

Answer 1

The information out there about PCI passthrough is terribly outdated and unhelpful.

I have an HD 7850, and a junk PCI vga-only Nvidia card that has to be a decade old. Neither of these devices support Function-Level Reset. They both work flawlessly being passed-through to DomUs; I run my Domain0 headless with no graphics card, both of my cards are sent to pciback at boot up via a kernel argument.

Xen is quite good about assigning resources to a VM and not allowing access to other resources; you can pin your vcpus to pcpus (cores, threads) and assign dedicated memory. Going with a high end Nvidia, you may have some work cut out for you- but it's hard to say since the papers/writeups that say that also say things like Function Level Reset is a necessity: it no longer is.

Answer 2

The critical part in your setup is 3D performance.

I've worked quite a bit with Xen 2-3 years ago - resource allocation worked fine back then and I'm sure that the technology has improved since. Everything but gaming either uses very few resources (file serving, streaming...) or isn't real-time critical (compiling on the dev system, etc).

I havn't done GPU passthrough, but cursory googling doesn't indicate that it works reliably, tweaking is required. Basically, it won't be plug&play - you are pretty much guaranteed that a few/some/many games won't work.

The most reliable option IMHO would be to use Win7 as a host and for games, and run the other services as virtual machines via VirtualPC or VirtualBox. Its not the most elegant solution (everything depends on Windows, every reboot means stopping all the VMs, etc.), but if you want gaming that Just Works, thats the way to go (other than using a second computer).

Answer 3

Is this feasible in general?

Yes. I have rencently performed Xen VGA passthrough and it worked fine. Take a look at the following video: http://www.youtube.com/watch?v=Gtmwnx-k2qg

Is the GPU passthrough usable enough with non-server GPUs?

Yes and no. Officially, only NVDIA Quadro FX 3800, 4800 and 5800 are supported, but I've successfully used a NVIDIA GeForce GTX 460 as in the previous video and Teo En Ming used a GeForce 8400 GS. It really depends on the graphic card.

When buying, what technologies should I look for on the motherboard (most likely choosing Gigabyte + Intel CPU again)

You should make sure the chipset support I/O MMU (AMD-Vi/VT-d), that the CPU support AMD-V/VT-x and that the graphic card support function level reset (FLR).

Note that this technique is not yet mature and it is not guaranteed to work until you try it!

Answer 4

Depending on the games you want to run, a better option may be to use Wine on top of Linux: check out the Wine Application Database, plus there are commercial versions of Wine that may provide better compatibility. You don't even need a Windows license.

In a way, Wine provides GPU pass-through, but it's at the Windows API level rather than at the PCI level. 3D performance is often comparable between Windows and Linux+Wine.

Answer 5

Based on my experience with Xen, here the Q/A:

1. Is this feasible in general? Yes, given the right hardware and configuration.
2. Is the GPU passthrough usable enough with non-server GPUs? Yes, there are plenty of user reports who use typically AMD cards for VGA passthru and gaming under Windows.
3. When buying, what technologies should I look for on the motherboard (most likely choosing Gigabyte + Intel CPU again)? VT-d (also called IOMMU or AMD-V by AMD) is an absolute must. With Intel CPUs both the CPU and the motherboard/BIOS must support VT-d. (Some old Intel chipset/CPU configurations also support VT-d, but that's not something you want to use for a modern system.) Look here: http://wiki.xen.org/wiki/VTdHowTo. In addition, make sure you got a tested and proven graphics card. Otherwise it may not work. See XenVGAPassthroughTestedAdapters for reference, though the list is not complete. Be very careful with the hardware selection and - if possible - buy from vendors who support Linux and specify VT-d / IOMMU in their technical specifications. Why? Because a BIOS update can easily brick your system. In any case, make sure which BIOS version supports VT-d, and don't upgrade unless you got a very good reason. Also, beware of vendors with no Linux experience - their answers can be misleading (to put it politely).
4. Is there some virtualization technology creeping on the horizon that is worth waiting for? Both KVM and VMware are making some efforts to reproduce what Xen is doing with VGA / PCI passthrough. Some KVM and VMware users have reported success with VGA passthru, more so with PCI passthru. I'm not familiar with Microsoft's Hyper-V, but I've seen reports that it also supports some sort of VGA passthru. I don't know how good Microsoft supports Linux on Hyper-V.
5. How good is Xen at dividing resources? It shines! At default settings it works great with balancing the CPU resources. It can be easily fine-tuned, but there really isn't much to do except reserving a fixed amount of memory for dom0 and perhaps giving it a little higher priority in the scheduler. The only weak point I could find is network configuration - it can be tricky and I still have to solve some minor issues.
6. The file-server and multimedia-streaming portion of the workstation has to work even if the game console part is eating 100% of its resources and vice-versa - A: I haven't seen problems with that, and in any event it can be tuned. Give dom0 enough resources.
7. The system will run on top of 8 disks with software Raid6 and LVM, should I buy a dedicated disk for the game console portion of the system? Xen works very well with LVM. However, in a Windows domU it's important to install the GPLPV disk driver to get good performance! You also have the option of passing through a SATA controller to the guest so it gets direct access to the hardware. If ~140-240MB/s file transfer speed between an SSD and striped LVM disks is acceptable, then I wouldn't bother adding a dedicated disk for gaming. Your RAID/LVM combo should do OK.
8. I have read about some problem with Xen and I/O performance - A: These reports may have been based on file based storage, which performs poor with other hypervisors too. Using LVM should provide good performance, as long as dom0 has sufficient RAM and CPU resources to handle I/O. So far I've never noticed any I/O problem, even under 100% CPU load of all cores (i7 3930K 6-core CPU).