hi, for a long time now, we've been running OmniOS and Nappit SANs on SuperMicro kit, so we have a virtual machine of OmniOS (with Napp-it for management ui) presenting iscsi storage back to the host via the standard Comstar components, so that we can roll our own converged storage devices.
We had been rock solid for more than a year on these under vSphere 5.
Since we upgraded to vsphere6 we started to see iscsi dropouts from the Comstar layer and eventually through trial and error we have identified the root cause of the problem to the be stability of the vmxnet3 adapter under OmniOS.
We have switched the nics on our vsans to use e1000 virtual nics and stability and throughput is back to where it always was.
I saw there was a recent announcement packet loss of vmxnet3 under vsphere6 and suspect - and I guess a related packet loss issue probably applies to openSolaris derivatives as well.
http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=203949...
I just hope this helps someone in the same boat and hope that VMware fix whatever bugs were introduced in vsphere 6..
Its an easy one to reproduce - take a commodity host - slap omnios on using hardware pass-through for the LSI controllers and present iscsi to the host via standard vmxnet 3 and then try multiple storage vmotions and you'll soon see the iscsi storage drop out even with all VAAI options disabled. As soon as you switch the vnic back to e1000 you'll see stability.
Also we noticed previously that when using vmxnet3 NICs on OmniOS the NIC order did not match the device order in VMware which created confusion (this issue had been around since vSphere 5 and possibly before). When we switched to e1000 adaptors, the order of the nics were presented was correct inside the OS.
Message was edited by: Ashley Watson - added information about nic order errors and vmxnet3.
