I can't seem to come to any sort of conclusion on what's going on here. I have successfully connected two separate ESXi 5.1 boxes with an NFS share on a FreeNAS box. The connection is easily established and I can browse, create folders, and upload/download to the datastore via NFS, but the speeds are horrendously slow (1KB/sec?!). This behavior is consistent between both ESXi boxes. However, there are other NFS shares connected to Ubuntu instances that experience normal/fast speeds.
I did some digging and logged some of the NFS errors on the ESXi boxes:
2013-03-02T10:08:02.571Z cpu0:2899)WARNING: NFS: 1990: Failed to get attributes (I/O error) 2013-03-02T10:08:02.571Z cpu0:2899)NFS: 2053: Failed to get object 36 7ec0861b b c711aa4 75e5ba38 b77930a 4000a 0 15ded1 0 0 0 0 0 :I/O error
2013-03-02T11:09:14.224Z cpu0:2050)NFS: 4544: Received nfsstat3 code 791 for the procedure 3 client 0x410015601d20 sock 0x4100180b1100 server's universal addres s 10.0.0.10.0.111 2013-03-02T11:09:14.340Z cpu0:2050)NFS: 4544: Received nfsstat3 code 2049 for th e procedure 3 client 0x410015601d20 sock 0x4100180b1100 server's universal addre ss 10.0.0.10.0.111 2013-03-02T11:09:14.356Z cpu0:2050)NFS: 4544: Received nfsstat3 code 2 for the p rocedure 3 client 0x410015602090 sock 0x410018036cd0 server's universal address 10.0.0.10.8.1
I also did a nfsstat on the FreeNAS box and found bunches entries in getattr:
Client Info: Rpc Counts: Getattr Setattr Lookup Readlink Read Write Create Remove 0 0 0 0 0 0 0 0 Rename Link Symlink Mkdir Rmdir Readdir RdirPlus Access 0 0 0 0 0 0 0 0 Mknod Fsstat Fsinfo PathConf Commit 0 0 0 0 0 Rpc Info: TimedOut Invalid X Replies Retries Requests 0 0 0 0 0 Cache Info: Attr Hits Misses Lkup Hits Misses BioR Hits Misses BioW Hits Misses 0 0 0 0 0 0 0 0 BioRLHits Misses BioD Hits Misses DirE Hits Misses Accs Hits Misses 0 0 0 0 0 0 0 0 Server Info: Getattr Setattr Lookup Readlink Read Write Create Remove 3781 60 163 0 4909 65837 8 6 Rename Link Symlink Mkdir Rmdir Readdir RdirPlus Access 0 0 0 0 0 0 43 1682 Mknod Fsstat Fsinfo PathConf Commit 0 68 4 0 0 Server Ret-Failed 119 Server Faults 0 Server Cache Stats: Inprog Idem Non-idem Misses 0 0 0 0 Server Write Gathering: WriteOps WriteRPC Opsaved 65821 65837 16
Again, connecting to the NFS share is fine...I know NFS is working properly on the FreeNAS box becuase other Ubuntu linux machines are able to transfer without speed issues at all. I'm trying to figure out what exactly to tweak and where - are settings in ESXi or in the NFS server wrong? It also seems that, since the behavior is consistent between the two separate ESXi boxes, that the NFS on the FreeNAS box just isn't tweaked right for ESXi to play nice.
interesting - from esxcfg-nics -l --> have you verified that the NICs are linked fine?
are there physical switches connecting to the freeNAS box? are the switch ports showing any errors -- is jump frames enabled on the vmknic?
whats the latency from vmkping like??
HTH
~Sai Garimella
Hi
Welcome to the communities.
Please check physical port or host NIC configuration.
Have you checked this isn't just the sync write issue w/ESXi and NFS? i.e. everything the esxi box issues is a sync write. or are you noticing the issue on reads as well?
*edit* removed my rambling reply with mistakes and replaced with below *edit*
More info about this behavior can be found at this link:
Sync writes, or: Why is my ESXi NFS so slow, and why is iSCSI faster? | FreeNAS Community
Hello,
I assume you're writing to a ZFS pool? If so, I have a fix for you. If it's UFS, this may still work, but it's untested.
FreeNAS is using FreeBSD, and the way I fixed slow NFS speeds in ESXi was by patching the NFS code to ignore O_SYNC commands from ESXi. Basically I'm making it more like iSCSI for speed, but with the ease of NFS.
Check out : Technical Musings: Speeding up FreeBSD's NFS on ZFS for ESX clients
That patch still applies for FreeBSD 10. I'm currently running through benchmarks in FreeBSD 10, and I'm eying cutting out the NFS cache subsystem as it's unnecessary duplicated overhead when you're working with a ZFS pool. Not sure how many people use FreeBSD/FreeNAS as their ESXi SAN technology, but there is a lot of performance to be had with the right code.
I've run 10+ machines in production for years with the patch, and have yet to experience any corruption issues.