Losing Connectivity to iSCSI and NAS

[lewisma]

I am using ESXi 4.0.0, 261974 build of the software. I have local storage and also an Iomega StorCenter ix4-200d which is supposed to be VMware supported. From this Iomega i have 2 100GB iSCSI targets and a NAS share. Several times a day the ESXi host loses connectivity with the Iomega device so the VM's that are running become unresponsive. The only way to get the connectivity back is to do a Refresh/Rescan on the Storage from the host using the vSphere client. Sometimes if you wait long enough it will come back on it's own. There is is no set pattern for this it's just very random, the 2 VM's that are running on the local storage do not experience any problems and remain totally accessible. I have checked everything from a network connectivity side of things and everything looks OK, i have 2 other hosts running on the same network using local storage only and these are also OK.

I have attached a log file containing the errors, is this pointing to some kind of I/O error, should i be looking at a better more performing storage box.

Any help would be appreciated.

Thanks

Lui

[weinstein5]

Can you provide a little information in regards to the physical you are using to connect ot the iomega device? Is it isolated to only storage traffic - NAS and iSCSI going over different network? How many Vms are you hosting on the iomega device?

Also Thread moved to ESXi 4 Forum

If you find this or any other answer useful please consider awarding points by marking the answer correct or helpful

[lewisma]

Hi and thanks for your reply, the esx host and the Iomega are on the same network, there are no restrictions in terms of traffic. I have 4 VM's that are using the Iomega and i have another 2 VM's using local storage. I was wandering if there was any kind of bottlenecks. The Iomega has dual NICS which are bonded.

Thanks

[lewisma]

I have been monitoring it this week. I have so far plugged the Iomega into 2 different network switches (HP & Cisco) at the moment it's plugged into a Cisco 2980. The problem is still there but i'm not seeing it as frequently as before when it was plugged into a D-Link switch. It's maybe losing the connectivity once an hour or so and sometimes longer periods of time. There is a Jumbo frames option on the Iomega configuration for Network Services.

The 2 options here are 4000 and 9000, at the moment i tried both settings but this dosesn't seem to make any difference in terms of losing the connection.

[DuncanArmstrong]

From the logs provided, it looks like you're facing some outright connection refusals at some stages.

Aug 27 11:09:29 vmkernel: 10:03:06:34.390 cpu0:4100)WARNING: NFS: 277: Lost connection to server 172.17.45.17 mount point /nfs/nfs-share-02, mounted as 93d0da44-ce50f1f1-0000-000000000000 ("nfs-share-02")
Aug 27 11:30:02 vmkernel: 10:03:27:07.576 cpu1:4100)WARNING: NFS: 277: Lost connection to server 172.17.45.17 mount point /nfs/nfs-share-02, mounted as 93d0da44-ce50f1f1-0000-000000000000 ("nfs-share-02")
Aug 27 13:15:28 vmkernel: 10:05:12:33.772 cpu1:4100)WARNING: NFS: 277: Lost connection to server 172.17.45.17 mount point /nfs/nfs-share-02, mounted as 93d0da44-ce50f1f1-0000-000000000000 ("nfs-share-02")
Aug 27 13:36:10 vmkernel: 10:05:33:15.941 cpu0:4100)WARNING: NFS: 277: Lost connection to server 172.17.45.17 mount point /nfs/nfs-share-02, mounted as 93d0da44-ce50f1f1-0000-000000000000 ("nfs-share-02")
Aug 27 13:56:52 vmkernel: 10:05:53:58.154 cpu3:4100)WARNING: NFS: 277: Lost connection to server 172.17.45.17 mount point /nfs/nfs-share-02, mounted as 93d0da44-ce50f1f1-0000-000000000000 ("nfs-share-02")
Aug 27 14:17:26 vmkernel: 10:06:14:31.423 cpu1:4100)WARNING: NFS: 277: Lost connection to server 172.17.45.17 mount point /nfs/nfs-share-02, mounted as 93d0da44-ce50f1f1-0000-000000000000 ("nfs-share-02")
Aug 27 14:38:17 vmkernel: 10:06:35:22.626 cpu0:4100)WARNING: NFS: 277: Lost connection to server 172.17.45.17 mount point /nfs/nfs-share-02, mounted as 93d0da44-ce50f1f1-0000-000000000000 ("nfs-share-02")
Aug 27 14:59:34 vmkernel: 10:06:56:39.894 cpu0:4100)WARNING: NFS: 277: Lost connection to server 172.17.45.17 mount point /nfs/nfs-share-02, mounted as 93d0da44-ce50f1f1-0000-000000000000 ("nfs-share-02")

Can you check for IP conflicts (specifically 172.17.45.17)?

How many target IPs are there to connect to, in case you're doing multi-pathing?

Do you have more NFS shares? The disconnections seem to apply to only nfs-share-02.

Do all ESX hosts lose connectivity at the same time?

Have you contacted IOmega yet?

[lewisma]

Duncan

Thanks for your reply, i have checked the following:-

No IP address conflicts

There is only target IP

I have 4 NFS shares but am currently only using share 02.

I have 3 ESXi hosts but the other two are using local storage only, the plan is to move these local VM's to the NAS but need to get this problem resolved before i do that. The other 2 hosts are fine but there not using any NAS. I haven't contacted Iomega yet, just made sure the device has the most up to date firmware, changed jumbo frames as a test, that's really it this point apart from googling the message. Hope this helps

[DuncanArmstrong]

Having a fresh mind after the weekend helps.

Here were the connection refusals; I attached different log snippets in my previous post.

Aug 27 11:35:59 iscsid: connect failed (111,Connection refused)

Aug 27 11:36:01 iscsid: connect failed (111,Connection refused)

Aug 27 11:36:01 iscsid: connect failed (111,Connection refused)

The messages log doesn't reveal much more; I unfortunately expect that the issue lies beyond the ESXi host. It doesn't look like EMC has certified the use of any Iomega NAS/SAN arrays for use with VMware ESXi. I'm not privvy on the details as of why not; you could inquire with EMC/Iomega to find out if there are problems, or if they simply have not completed the process. Single controller support is possible when used with ESX, and with firmware 2.1 and above.

Sorry I can't be of much more help; I'd suggest working with EMC/Iomega and giving VMware Technical Support a call if it's deemed an ESX host issue.