4

I'm configuring software that will need about 12 cores to run, but I am running it on a vSphere virtual machine, meaning that at most 8 (virtual) cores can be configured.[1] The physical machine that this will be running on has 24 cores available, and hosts only one other virtual machine. That VM has 8 cores allocated as well.

Since my machine has 8 cores, and the other machine has 8 cores, this means that 8 cores seem to be left completely idle. This seems wrong.

I'm guessing that VMWare has done something clever -- That even though I can only have 8 cores allocated to my machine, in reality, if there are 24 cores on the back end, then I am guaranteed to get 8 of them, but can use 24 of them if no other machine is using them.

I've been reading about co-sharing in vSphere,[2] but it's a bit over my head.

Can anybody explain how this works?

Edit: This is the explanation I've been given about the 8 CPU limit, but I need to confirm it.

vSphere uses dynamic processor load balancing that allows each assigned core in a guest access to all cores on the host. While the guest OS will only see 8 physical processors, each processor has access to a pool of 24 cores. This is very similar to how a mainframe works.

This hints at the 8 cores behaving more like 24 cores, but now this just seems wrong. Is it?

References:

Ben Pilbrow
  • 11,995
  • 5
  • 35
  • 57
mlissner
  • 990
  • 3
  • 8
  • 17

3 Answers3

9

What this means is that you'll never have more than 8 threads executing in parallel in your virtual machine. However, through the magic of ESX resource allocation you can give those threads quite a bit of horsepower. You're not limited to the max-rate of your actual CPUs, ESX's CPU load-balancing methods will permit running faster than that... so long as what you're doing with it is able to do such slightly out of order execution.

This is accomplished by leveraging a queue structure for CPU resources. Work is dispatched to multiple processors as needed. A single vCPU may execute on any of the physical CPUs on the system (or the CPUs in the local-node if in a NUMA system). Achieving vCPU performance in excess of physical CPU performance is done by dispatching work from a single vCPU to multiple physical CPUs in parallel or at least very close together.

When it comes time to reassemble work returned by multiple CPUs to emulate a single vCPU, it does look to the VM like a single CPU working very fast. ESX reassembles the multiple work-units into the correct order.

Not all workloads are well suited to this, of course. Jobs that submit a lot of iterative work that is loosely related to each other if at all is perhaps the best case for this. Jobs that involve lots of tight dependencies with earlier instructions, such as with the recursive calls of crypto algorithms, won't be able to scale nearly as far.

sysadmin1138
  • 131,083
  • 18
  • 173
  • 296
3

vSphere 4.0 has a hard limit of 8 vCPUs. That's all it will use, and it's a fairly sensible limit. vSphere is more designed to handle many guest VMs on a host, rather than just a couple. You may want to see if you can cluster 3-6 VMs instead of just doing 2.

Hyppy
  • 15,458
  • 1
  • 37
  • 59
  • Does VMWare have a different product that can handle more than 8 cores per guest? Why the limitation if not for money? – mlissner Apr 19 '11 at 01:08
  • @mlissner 8-cores is as multi-threaded as it gets for vMWare-ESX right now. That's Enterprise-Plus. The next versions may allow more. – sysadmin1138 Apr 19 '11 at 01:13
  • 3
    @mlissner It's not a limit imposed because they want to make more money, it's there for technical reasons. Scaling out the low-level parts of virtualization is not a simple task. They have increased vCPU limits dramatically over the last few versions. 2.0 was limited to 2, 3.5 allowed 4, and now 4.0 allows 8. For performance and scalability reasons, 8 is a good limit right now. We generally only allow 4 as a hard limit unless someone can demonstrate a dire need for 8 due to the way that the hypervisor implements SMP for guests. – Hyppy Apr 19 '11 at 02:51
  • 1
    Agree with this: As much as I love virtualisation, sometimes it isn't the answer. If you need *lots* of cores per server then that is one of the times when it probably isn't the answer. – Rob Moir Apr 19 '11 at 10:50
  • Quite a hard limit. MS just raised their equivalent to 32 virtual cores. I am quite happy with the performance of some 12 core virtual machines. – TomTom Sep 25 '12 at 16:06
-1

This has to do with the way that they license their software. You get so much power for free. If you want more than that, you have to pay for it. http://www.vmware.com/download/eula/multicore.html

Nixphoe
  • 4,524
  • 7
  • 32
  • 51