I've been using VMXNET3 for my network device and recently implemented SR-IOV. I've noticed a huge drop in performance.
Host is SuperMicro X9DRH-7F, 2x E5-2667v2, nic is x520-DA1 connected to my switch via DAC.
I don't know what my max output is across the physical network as I don't have any other 10Gb clients. iperf and ping show good 1Gb performance.
Using iperf between guests on the same host with VMXNET3 I get a little over 6Gbps.
Switching to SR-IOV, iperf from FreeBSD server dropped as low as 3Gbps to my windows VM and 4.8Gbps to my Ubuntu VM.
Doing a loopback test with iperf I get 40+Gbps on the FreeBSD (FreeNAS) VM, 9.5Gbps on the windows VM and 25Gbps on the Ubuntu VM.
I've tried TSO and LRO, a number of different combinations of sysctl tuneables and the changes are very little if anything at all. Only switching back to VMXNET3 gets me back to 6Gbps.
I would appreciate some help narrowing this issue down. I'm hoping to achieve 9+Gbps, or even more between guests on the same host.
The speed of your physical NIC and underlying physical network are irrelevant if the VMs are connected to the same port group in your vSwitch. The packets between those VMs never touch the driver for the physical NIC, never mind the hardware.
I understood that was the case for vmxnet3, but i thought sr-iov exposed the nic hardware directly to the vm bypassing the host kernel?
Either way, any suggestions for tracking down the bottleneck?
OP, did you ever discover more regarding your issue? I suspect processor P-states/C-states in BIOS might be causing your poor performance.