0

Good day all! I will try and make this short and sweet. My current setup is:

XS708T NETGEAR 10gbE switch

ESXi Machine #1:

  • 2x Intel 10gbE
  • pfSense (VM) with WAN on one vSwitch, LAN on other vSwitch, NIC's above respectively for each.
  • pfSense, Ubuntu Server (x2), FreeNAS server, Windows Server VM's

ESXi Machine #2:

  • 1x Intel 10gb (SFP+)
  • Ubuntu Server (x2), FreeNAS server, Windows Server VM's

VLAN's: set up on pfSense and switch

  • 10 = VM's
  • 20 = LAN_OTHER
  • 30 = WIFI
  • 99 = ADMIN

So here is the issue: using iperf between both Ubuntu VM's on separate AND same ESXi machines are able to reach the full 10GB/s throughput when on the same VLAN. I have ran in both directions as well. Watching systat -ifstat on pfSense, I can verify no traffic is traveling through pfSense. If I try and go between two Ubuntu VM's on the SAME or DIFFERENT ESXi machines, but with different VLAN's (ie: 10 & 99), It drops down to about 2 - 3 GB/s throughput. During this test, I can see all the traffic traveling through pfSense.

So my thought process is this; my switch has been verified capable of handing the 10GB/s throughput when on the same VLAN. I have monitored pfSense during all the test to verify that I'm not hitting any sort of hardware (CPU) limits.

So is this an issue with my setup, or possibly my pfSense VM? Any and all help is appreciated! Thank you!

Rex
  • 7,815
  • 3
  • 28
  • 44
Matthew Toye
  • 3
  • 3
  • How have you sized your pfSense VM? This web site suggests that getting 10 Gb of throughput may require a decent amount of horsepower (8 cores?): https://www.firewallhardware.it/en/pfsense_selection_and_sizing.html – Todd Wilcox Nov 14 '17 at 14:35
  • Hmm.. I will give that a try today for sure. I'll let you know! Current size is 2vCpus, but I'm not even close to maxing out CPU usage. – Matthew Toye Nov 14 '17 at 16:23

1 Answers1

2

The pfSense VM has quite a bit of overhead to fight with - ingress packets go through the host NIC, host vSwitch, are copied to the VM, pass the guest stack, are processed and egress packets take the same way back. All this happens in software and requires quite a few context switches. These cause latency on the packet and - through the resulting bandwidth-delay product - may limit the overall throughput for a delay-sensitive transport protocol such as TCP.

There are three basic approaches: reduce latency (reserve host resources for VM, use a paravirtualized vNIC or a dedicated NIC with passthrough, fine-tune NUMA setup), increase the TCP receive window with TCP window scaling, or use multiple TCP flows at the same time.

Zac67
  • 8,639
  • 2
  • 10
  • 28
  • This makes sense for sure. I think the easiest way may be to add a dedicated supported nic in passthrough to the VM. When referring to TCP window scaling or multiple TCP flows, are you referring to using flow control on the vSwitch in ESXI? Also , multiple TCP flows as in extra vNics? Thank you very much for your answer and patients! – Matthew Toye Nov 14 '17 at 22:34
  • The [TCP window scale option](https://en.wikipedia.org/wiki/TCP_window_scale_option) has nothing to do with Ethernet flow control. Rather, it's a way to get around the "[long fat network](https://en.wikipedia.org/wiki/Long_fat_network)" problem. Multiple TCP flows are just that: if transmitting a single file can't be accelerated, transmit two or more at the same time. – Zac67 Nov 15 '17 at 07:23