VMware ESXi and ZFS hardware & configuration recommendations

Question

I want to finally spend a little money upgrading my company's virtualization infrastructure, but I've been looking at hardware for a couple of weeks now and I'm not quite sure what's appropriate for our needs. In addition to buying some actual server hardware, I want to replace our current VMware Server with VMware ESXi. The initial budget is probably somewhere around $2000 but may be able to eventually scale up to $3000 if we need to tweak the performance later.

After reading some scary articles on RAM errors and hard drive failure rates over on StorageReview (and after dealing with several disk failures myself), I want to move to ECC RAM and I want to store our VMs on redundant ZFS volumes to protect against silent data corruption and data loss.

Note: I know there are already various questions about VMware ESXi and ZFS hardware recommendations, but I couldn't find any discussions about the performance of VMware ESXi with the VMs stored on ZFS.

Current configuration

• VMware Server on Windows 2003
• Athlon X2
• 8 GB RAM
• 2x250 GB SATA mirror (dynamic disk)
• 2x640 GB SATA mirror (dynamic disk)

Right now we run 4 VMs 24/7 on an Athlon X2 with 8 GB RAM. Occasionally we need to fire up an additional VM or two, and the performance suffers. We also want to virtualize a couple additional servers, but I don't see us needing more than a dozen anytime in the next couple of years. I've read that VMware ESXi scales better since it's a bare-metal hypervisor and it has various performance optimizations.

VMs:

• 2 GB RAM, Win2003 (running Fogbugz, including MS SQL Server)
• 512 MB RAM, Fedora Linux (running legacy systems such as MediaWiki, RT, and Bugzilla)
• 2 GB RAM, WinXP (regularly used for software development & testing)
• 1 GB RAM, WinXP 64-bit (only occasionally used for testing)
• 1 GB RAM, WinXP Japanese (only occasionally used for testing)
• 1 GB RAM, CentOS Linux (regularly used for software development & testing)

Planned configuration

ESXi server:

• Entry-level Dell or HP quad-core Xeon server
• Dual quad-core Xeon 55xx (one CPU initially--is Nehalem worth the extra money?)
• 8-16 GB ECC RAM (max capacity of at least 32 GB RAM)

I figure that will set me back $1000-$1500 if I buy the RAM separately from NewEgg, but I'm not sure if it's better to put more money toward a newer-generation Xeon, if I should put it toward a second CPU, or if I should just save it for now.

Additional VMs:

• 1.5 GB RAM, Windows 2003 Small Business Server (currently only used as the domain controller; will probably be rebuilt/replaced soon)
• 2 GB RAM, Windows 2003 Server (infrequently-used Terminal Server, along with various license servers for different parts of our development environment)
• A few other Windows and Linux workstation VMs used occasionally for testing

ZFS server:

I figure I could repurpose the current Athlon X2 server as the ZFS server by installing one of the free Solaris derivatives. (Thanks MarkM for pointing out that the OpenSolaris group disbanded after Oracle discontinued their support a couple weeks ago--I'll have to follow the progress of Illumos' open-source fork). I'm planning to make the storage available to ESXi via iSCSI over a direct gigabit Ethernet link.

The motherboard supports 8 SATA drives, and I figured I would add an additional 4- or 8-drive eSATA enclosure so I can have several ZFS mirrors or raidz2 volumes.

I'll probably replace the memory with ECC RAM for data integrity, and might add SSDs for the ZIL/L2ARC to improve ZFS performance, but I'm not sure if the money spent on SSDs would be better spent on SAS drives instead.

Any suggestions?

Given these specs, am I sizing the servers correctly? Will a mix of SSD and consumer/"enterprise"-grade 7200 RPN SATA drives give us sufficient disk performance, or do we really, really need to upgrade to SAS? Are there any gotchas I need to watch out for? Am I missing something important?

Most importantly, has anyone had experience setting up a similar configuration--and, if so, how did it perform?

Answer 1

Like notpeter said, you're on the right track for a low budget setup.

My advice to you (learned this the hard way):

• Don't put all your eggs in one basket. You should seriously consider having an extra ESXi server in case you get hardware errors - wich virtualization is all about
• Buy lots and lots of RAM. VMs perform better, and your storage is less busy writing/reading swapfiles for guests.
• Unless you have some absolute requirements for IOPS, a low-budget SATA 7200 rpm setup will work just fine

Answer 2

Since OpenSolaris is dead, Solaris 11 is 6+months out and Illumos is still a work in progress, you should consider Nexenta Core platform (nexenta.org not nexenta.com). Solaris kernel plus debian (apt) userland is quick to learn if you're already familiar with Ubuntu or Debian Linux. If you decided to switch to Solaris 11/FreeBSD down the road, you can bring your ZFS pools along for the ride. I started with ZFS-Fuse under Linux, migrated to OpenSolaris and then onto Nexenta:CP all with the same pool.

My setup at home is similar to your suggested ZFS storage server. I find a dual core @ 2.8Ghz (Core i5-650), 8GB ECC Ram and a 4x2TB (WD RE4 7200RPM) mirrored zpool with no SSDs quite performant. ESXi connects via NFS or iSCSI, while Win/Mac clients connect via CIFS. I am easily able to saturate a single gigabit connection (~95MB/sec) and the disks have even better throughput locally. I honestly don't have sufficient network gear or IOPS load at home to see how much latency would benefit from having SSDs for ZIL and L2Arc, but will soon be adding more ram and an SSD for L2Arc to see if I can really get deduplication working quickly.

I think given enough ram (8GB+), a decent SAS card and mirrored pools your Athlon X2 box is likely up to the task. IMHO money spent on 10k/15k SAS is a waste for ZFS unless you plan to leverage multipathing/smart SAS enclosures or are stuck with only 4-6 drive bays. You mention eSATA; consider multilane SAS (4x) as an alternative for enclosure connectivity. Unless you were planning on having 1:1 esata ports to drives, you'll need 4to1/5to1 eSATA port multipliers which multiplex multiple drives over one channel. Multilane SAS (SFF 8088/SFF-8470) is four true SAS/SATA channels per cable, with breakout cables on one/both ends depending on your cards and enclosures. LSI SAS non RAID cards are great for ZFS even the high end $500 LSI SAS 9200-16e (Quad 6Gbps SAS 4x ports) is supported under Nexenta.

As for the ESXi box, any big vendor 4/6 core server will meet your needs today, but don't count on being able to fill the 2nd processor socket down the road, upgrade pricing may not be reasonable or even available (I got burned by this). If you're going do whitebox ESXi (2nd box for DR?) the biggest gotcha is making sure you have a NIC on the ESXi HCL. Intel server NICs will likely to work, but many onboard Intel e1000 and RealTek will not. ESXi won't even boot/install unless it finds a supported NIC. Cheapest PCIe addon NIC supported is the $30 PCIe x1 Intel Gigabit Ct Desktop (EXPI9301ct), but if this is your primary esx box you should definitely pick up a higher end multiport Intel NIC, potentially with iSCSI TOE. Sadly AFAIK, no TOE support at all on the ZFS side (OpenSolaris/Nexenta).

Honestly, I think you're on the right track. Building a poor mans ZFS SAN for ESXi will likely suit your needs just fine and give you some room to grow, especially if you stick with mirrored (not RAIDZ) pools. Once your IO needs exceed 2xGigE or you get a 2nd ESXi host, things get trickier (L3 switches, 10gigE, 4Gig FC, etc) but you'll cross that bridge when you come to it. Oh yeah, and don't forget even though ZFS is quite stable and snapshots are powerful, it's no substitute for proper backups.

btw, thanks for linking to my earlier answer in your question.