7

I am looking to upgrade one of our servers, an 8 core server running VMware Server 2. The VMs it runs are mostly web, file, and email servers; specifically, it runs three webservers, 2 email/file servers, and a few Jabber/XMPP Servers.

When we originally set this machine up, we had the two web servers configured with two vCPUs. We had some very serious performance issues. At the time we were only running 3 VMs with a total of 5 vCPUs. We were also running VMware Server 1 and a RAID1. We have now upgraded to a RAID10 and I would never consider anything less! :-)

The new server will have VMware ESXi on it instead of VMware Server. We are looking at either another 8-core machine with faster processors, or a 16- or 32-core server. I want to make sure that I avoid the issues we had before.

Because I had such issues with multiple vCPUs before, my plan was to replace the primary web server VM with multiple identical webserver VMs (2 - 4), each with a single vCPU, serving content from an NFS share, and place these in a load-balanced configuration. SO I would basically be replacing one VM with at least 4: 2 small webserver VMs, a file server VM, and a load balancing VM. My plan here being twofold: First, I can avoid any multiple vCPU isuses I saw before, and secondly, I can spawn new web server VMs to handle increased load.

Note that I understand I am not getting all the benefits of the load balancing because I still have a single point of failure.

Is this a good plan, or is it unnecessary? Can VMware ESXi handle multiple vCPUs better than VMware Server 1 could over three years ago? Which will perform better, a single large webserver VM with 2 - 4 vCPUs, or 4 VMs with a single vCPU, in a load-balanced configuration?

Josh
  • 9,001
  • 27
  • 78
  • 124
  • As you probably aware, VMware suggests using 1vCPU per VM, unless you have a software that will actually not only benefit but can actually utilize multiple CPUs. – Vick Vega Mar 01 '11 at 22:22
  • 1
    @Vick, where does it recommend that? isn't that old news? – Chopper3 Mar 01 '11 at 22:37
  • That's exactly what I wanted to know @Vick. As far as I could tell, the docs that said that were years old. – Josh Mar 01 '11 at 22:48
  • 1
    @Chopper3 - Top of page 8 - http://www.vmware.com/pdf/exchange_best_practices.pdf (yes it is exchange specific, but it was published in 2010.) – Clint Mar 01 '11 at 23:25
  • 1
    @Chopper3♦, There's a lot of talking on the web about this topic, dozens of pages of white papers and discussions which come with the conclusion I stated. Look for "recommended vcpu per vm" in google. There's a lot of material about it. – Vick Vega Mar 02 '11 at 00:40
  • Thanks guys, I really didn't think that was still a valid recommendation but I guess I'm wrong, thanks again. – Chopper3 Mar 02 '11 at 09:38

2 Answers2

7

Can VMware ESXi handle multiple vCPUs better than VMware Server 1 could over three years ago?

Yes, they're totally seperate product lines, and ESXi can handle multiple vCPUs very well. I'm running a 5 node vmware farm right now with a mix of machines on each box, each with 1, 2 and in some cases 4 processors as needed and it works very well indeed.

Which will perform better, a single large webserver VM with 2 - 4 vCPUs, or 4 VMs with a single vCPU, in a load-balanced configuration?

That's less of a VMWare question and more of a application/website testing question. VMWare (or any other modern bare metal virtual machine) can do either way - there might be a cost to having lots of vCPUs attached to one guest but this might be offset by the resource costs of having multiple guests running and by performance issues (or improvements, for that matter) your web application might realise from a more distributed approach.

On that last issue, its really a matter of testing and architectural preferences rather than the one true way, but my general feeling is that if you can distribute the application over more than one virtual machine easily, its better to build a few smaller virtual machines than one big one. Gives you more flexibility and helps performance in the long run as you can move things about if you have to. In other words, scaling outwards rather than upwards

With physical machines I think we think of scaling upwards, putting as much into one OS "instance" as possible to get the most out of the cost of the licence and of the hardware, but with virtual machines those equations change a bit - if you can get more than one virtual OS instance for the price of a licence then this might change the way your server infrastructure is designed.

edit

http://blog.peacon.co.uk/understanding-the-vcpu/ is an article Iain mentioned while discussing this question in chat that explains in some detail why its important to think about vCPU allocation when creating your VMs.

Rob Moir
  • 31,664
  • 6
  • 58
  • 86
  • 1
    Definitely its something to be tested. But VMware still recommends single vCPUs over more, as the hypervisor must schedule two simultaneous free cpus instead of one to schedule for the VM. That creates more overhead. – Steven Mar 01 '11 at 22:38
  • 1
    Agreed on the vCPU count. Recent ESX/ESXi versions have gotten much more scalable, the vCPU per physical core limit has gone up from 8 (in 3.5) to 25 (in 4.1). Not that it's wise to hit the limit (the core you're splitting 25 different ways is still just one core). If your CPU resources aren't in contention, then it can definitely be worthwhile to have extra compute resources available to VMs under heavier load. – Shane Madden Mar 01 '11 at 22:40
  • @Steven - absolutely agree, its a balancing act, I think. You have to balance any requirement for raw power against a loss of flexibility if you're going to put a lot of guests on the host. – Rob Moir Mar 01 '11 at 22:46
  • 1
    Just revisiting this question, a year after we made the switch. **VMWare ESXi is indeed running many VMs with multiple vCPUs with no issues whatsoever**. We did go with a 32-core host and have multiple VMs with 2 and 4 vCPUs each, all running great. The improved performance of ESXi over VMWare Server blew me away. No issues with ESXi at all, it's working great. – Josh Mar 22 '12 at 15:25
4

I can tackle this question in a number of ways, but I'll start of with horizontal vs. vertical scaling.

Horizonal scaling

Pros

  • Can scale very large
  • Can scale across hosts (ie a vmware cluster)
  • Fault tolerant
  • Low/No downtime during maintenance

Cons

  • More overhead from VM OSes and Load Balancers
  • More maintenance/management (eg patching, SSL certs)
  • More website version management across web servers
  • Additional testing and development may be required for web site behind load balancer
  • Licensing?

Vertical Scaling

Pros

  • Scales easier (just add another vCPU or two, or more memory)
  • Only one web server to manage, license, etc.

Cons

  • Downtime during maintenance/upgrades
  • Scaling too big can cause performance issues in a VM.
  • Doesn't truly scale.

So, both have their goods and bads, and I am a big fan of horizontal scaling, but here's what I'd recommend to you:

  1. Scale vertically to a certain point (I'll get to that certain point in a minute).
  2. Do your homework on what it is going to cost time and money wise to get going horizontally.
  3. Give yourself plenty of time to test your horizontal scaling solution.
  4. When you've reached the point where it's time to scale horizontally, you can ramp back down your vertically scaled server, make some clones, throw them behind your already configured load balancer, and watch it rock and roll.

Okay, so what is that certain vertical scaling point?

It depends on your hardware.

VMware has done a lot to vCPU scheduling since ESX 2.x, and there is more room to grow vertically. How much really depends on the CPUs you get.

Sizing VMs for performance nowadays requires understanding of NUMA (Non-Uniform Memory Access. Basically, you don't want a VM to be bigger than your CPUs NUMA node. Here are some links to help guide you in that area:

http://frankdenneman.nl/2010/02/sizing-vms-and-numa-nodes/

http://frankdenneman.nl/2010/09/esx-4-1-numa-scheduling/

http://frankdenneman.nl/2011/01/amd-magny-cours-and-esx/

I guess I'll wrap it up by saying that CPU has never been too much of an issue for me, unless you are oversubscribing CPU on your host. Instead, memory is normally the culprit. Ballooning/swapping memory os a very noticeable performance issue.

Good luck!

Community
  • 1
JakeRobinson
  • 2,886
  • 17
  • 26