VMXNET3 performance on Linux on ESX 5.0

Question

What kind of performance can I expect from VMXNET3 adapter when running two Linux guests on the same VMware ESX private network defined between those guests? I use VMXNET3 adapter for communicating between these OSs and E1000 adapter to talk to external world.

When I run iperf TCP performance test I get 3.7 Gbits/sec on the VMXNET3 adapter and 1.6 Gbits/sec on the E1000 adapter. I would have expected VMXNET3 to be up in the 10+ Gbits/sec range.

OS guests are powered by Red Hat Enterprise Linux 6.5, 64 bit, kernel 2.6.32-431.el6.x86_64 under the control of VMware vCenter Server Hypervisor 5.0.0 build 913577. Host server used for benchmarks is IBM x3950 M2 server model 7233-AC1, 24 cores Intel Zeon X7440 2.66 GHz, 256 GB RAM with 1GB Ethernet NICs.

VMware tools installed in guest OSs are v8.6.10 build 913593 (seems to be the latest at the moment).

score 8 · Accepted Answer · edited Apr 13 '17 at 12:14

8

That's a really old version of VMware ESXi. The performance does seem to get better with newer revisions, but what you are seeing is probably accurate for that vintage.

Is there any reason your VMware environment hasn't been patched or updated? VMware tools are usually tied to the host version and patches, so you're running 2012-era VMware tools on a 2012-vintage ESXi installation. I have tools version 9.4.0.

You can update the tools independently of the ESXi installation by leveraging the VMware Operating System-Specific Packages (OSPs).

See: How do I make sure VMware Tools is installed and running?

On a modern ESXi 5.5 local copy (same host) between vmxnet3 VMs running EL6.5, I see:

Client:

[root@xt ~]# iperf -t 30 -c mccartney
------------------------------------------------------------
Client connecting to mccartney_mirror, TCP port 5001
TCP window size: 23.2 KByte (default)
------------------------------------------------------------
[  3] local 172.16.2.12 port 35543 connected with 172.16.2.159 port 5001
[ ID] Interval       Transfer     Bandwidth
[  3]  0.0-30.0 sec  90.1 GBytes  25.8 Gbits/sec

Server:

[root@McCartney ~]# iperf -s
------------------------------------------------------------
Server listening on TCP port 5001
TCP window size: 85.3 KByte (default)
------------------------------------------------------------
[  4] local 172.16.2.159 port 5001 connected with 172.16.2.12 port 35543
[ ID] Interval       Transfer     Bandwidth
[  4]  0.0-30.0 sec  90.1 GBytes  25.8 Gbits/sec

edited Apr 13 '17 at 12:14

Community

1

answered Mar 07 '14 at 15:04

ewwhite

194,921
91
434
799

2

That's about what we get too btw – Chopper3 Mar 07 '14 at 15:06
Thanks for the reply. I have no control over the version of VMware being used. I was fiddling with all kinds of RHEL TCP settings in sysctl.conf, but none of it made positive difference (althought I was able to reduce performance by changing those settings). – Roman Kharkovski Mar 07 '14 at 15:09
@romankhar You may not have control over the VMware being used, but it's not reasonable to use such an old version without any patches or improvements. If changing this is not possible, at least try to use the [VMware OSP drivers and VMware tools](https://www.vmware.com/support/packages) inside of your guests. – ewwhite Mar 07 '14 at 15:11
@ewwhite VMware docs are confusing as to what Tools version can be installed into which ESX server. Here it says I can only install tools of 5.1 into 5.0 server: http://pubs.vmware.com/vsphere-55/index.jsp?topic=%2Fcom.vmware.vsphere.vm_admin.doc%2FGUID-A2491004-1C67-4E14-B47B-807E20C19108.html – Roman Kharkovski Mar 07 '14 at 17:36
@ewwhite From the very beginning I have enabled the checkbox on the VM properties to install latest version of Tools into my VM - I assumed that was done automatically. Or is that only latest for that specific ESX server? But not latest in a sense that website would have even newer version? If so, after having read VMware docs, I am still confused as to - can I install tools from ESX 5.5 into VM running under ESX 5.0? – Roman Kharkovski Mar 07 '14 at 17:45
@romankhar Sure, you can use newer VMware tools on older systems. But the bigger point is that the host (and even your RHEL 6.5 kernel) should be updated. – ewwhite Mar 07 '14 at 17:47
last night I upgraded both the ESX server to v5.5.0 build 1623387 and vmware tools in my Linux VM to the level 9.4.5.30548 (build-1598834). The VMXNET 3 performance has not changed one bit - still can't get more than 3.5 Gbit/sec between two VMs running on the same host. Perhaps the problem is something in the hardware itself being about 5 yrs old and something in VMXNET3 not working well on that hardware? – Roman Kharkovski Mar 18 '14 at 13:18
1

@romankhar I'm sorry. I've retired all of my systems of that vintage. They've been eclipsed by: Nehalem, Westmere, Sandy Bridge and Ivy Bridge CPUs. So your older systems are a few generations behind what's current. I don't have anything of that age to repeat the test with. – ewwhite Mar 18 '14 at 13:31

score 1 · Answer 2 · edited Mar 10 '14 at 10:37

1

Try changing ethernet driver settings.

/sbin/ethtool --offload eth0 gso off tso off sg off gro off

See

https://access.redhat.com/site/documentation/en-US/Red_Hat_Enterprise_Linux/6/html/Virtualization_Host_Configuration_and_Guest_Installation_Guide/ch10s04.html

If you experience low performance with the para-virtualized network drivers, verify the setting for the GSO and TSO features on the host system. The para-virtualized network drivers require that the GSO and TSO options are disabled for optimal performance."

This is about virtio drivers, but I think the same might apply to vmx drivers.

You can also find a lot of similar articles about VMWare: https://info.pleasantsolutions.com/Blog/VMware_Networking_Speed_Issue

edited Mar 10 '14 at 10:37

Jenny D

27,358
21
74
110

answered Mar 09 '14 at 11:30

Lari Hotari

171
6

1

This should not be necessary. The VMware results posted above were without any modifications to sysctl.conf parameters. – ewwhite Mar 09 '14 at 14:20
@ewwhite, you never know without testing. See http://kb.vmware.com/kb/1027511 http://kb.vmware.com/kb/2032709 – Lari Hotari Mar 09 '14 at 15:20
http://kb.vmware.com/kb/2030927 – Lari Hotari Mar 09 '14 at 16:05
1

I understand what you're saying about testing in general. However, those knowledge base articles don't apply to the original poster's environment. – ewwhite Mar 09 '14 at 16:07
Agree, but that doesn't mean you shouldn't try testing with offload settings disabled. – Lari Hotari Mar 09 '14 at 17:09
https://info.pleasantsolutions.com/Blog/VMware_Networking_Speed_Issue – Lari Hotari Mar 10 '14 at 07:13
http://infotechconsulting.co.uk/hardware/disable-tcp-chimney-offload-2/ – Lari Hotari Mar 10 '14 at 07:14
"vCenter Server Heartbeat is incompatible with TOE, a common feature of 10GB Ethernet cards." http://kb.vmware.com/kb/1024066 – Lari Hotari Mar 10 '14 at 07:15
http://kb.vmware.com/kb/2008925 – Lari Hotari Mar 10 '14 at 07:17
http://kb.vmware.com/kb/1009517 – Lari Hotari Mar 10 '14 at 07:18
http://kb.vmware.com/kb/1004337 – Lari Hotari Mar 10 '14 at 07:19
@ewwhite There are some more knowledge base articles related to Ethernet TCP offload settings. Some even have ESXi 5.0/5.1/5.5 in the product list, so I'm quite confident that it makes sense to test different offload settings (on both the guest and host, if that's possible). – Lari Hotari Mar 10 '14 at 07:21
Setting GSO and TSO to OFF made iperf speed tests slower, so I changed it back to ON. Still the best I get is 3.4 GBits/sec, which seems to be too slow. Perhaps the problem is indeed with the VMware tools version that needs to be updated... – Roman Kharkovski Mar 17 '14 at 17:07

VMXNET3 performance on Linux on ESX 5.0

2 Answers2

Linked