Hi Community,
When I use the vSphereClient and make a direct connection to the ESXi machines I can browse the NAS datastore without problems. But when I connect to the vCenter VM and try to use/browse the NAS Datastore it fails with a "Timeout" message. I tried to remove the NAS datastore on the ESXi machine when I have a direct connection to the ESXi machines.
Then I tried to create a NFS datastore via vCenter.
[2011-04-15 14:58:08.120 03008 error 'App' opID=110E0D33-000000B1] [WaitForDatastore] Timed-out while waiting to get datastore netfs://10.210.0.84//ifs/data/PS_Customers/200107/ information from host
[2011-04-15 14:58:08.195 03008 error 'App' opID=110E0D33-000000B1] vmodl.fault.SystemError
------ In-memory logs end --------
[2011-04-15 14:58:09.686 03008 info 'App' opID=110E0D33-000000B1] [VpxLRO] -- FINISH task-182 -- host-136 -- vim.host.DatastoreSystem.createNasDatastore -- 2087A1D6-4CB3-4E11-AD75-EBE50348F69F(A15B1CB4-60C5-44B5-BCBB-58CE825298B4)
[2011-04-15 14:58:09.686 03008 info 'App' opID=110E0D33-000000B1] [VpxLRO] -- ERROR task-182 -- host-136 -- vim.host.DatastoreSystem.createNasDatastore: vmodl.fault.SystemError:
Result:
(vmodl.fault.SystemError) {
dynamicType = <unset>,
faultCause = (vmodl.MethodFault) null,
reason = "Timed-out while waiting to get datastore information from host",
msg = "",
}
Args:
Arg spec:
(vim.host.NasVolume.Specification) {
dynamicType = <unset>,
remoteHost = "storage1.plusserver.de",
remotePath = "/ifs/data/PS_Customers/200107",
localPath = "Test",
accessMode = "readWrite",
type = "nfs",
userName = <unset>,
password = <unset>,
}
Anyway, vCenter shows the datastore, see pictures in section 1. But it's not "really accessible" , see pictures in section 2.
1)
2)
Service-Console View:
~ # esxcfg-route -l
VMkernel Routes:
Network Netmask Gateway Interface
10.254.10.240 255.255.255.240 Local Subnet vmk2
62.75.232.0 255.255.255.128 Local Subnet vmk0
10.1.1.0 255.255.255.0 Local Subnet vmk1
10.210.0.0 255.255.255.0 10.254.10.241 vmk2
default 0.0.0.0 62.75.232.1 vmk0
~ # ls -lha /vmfs/
/vmfs/devices/ /vmfs/volumes/
~ # ls -lha /vmfs/volumes/Isilon/
drwxr-xr-x 1 root root 56 Apr 8 10:16 .
drwxr-xr-x 1 root root 512 Apr 15 13:19 ..
dr-xr-xr-x 1 root root 0 Apr 15 02:00 .snapshot
drwxr-xr-x 1 root root 143 Apr 13 14:38 ISO Images
-rw-r--r-- 1 root root 18.0k Mar 23 13:47 index.html
~ #
~ # esxcfg-vmknic -l
Interface Port Group/DVPort IP Family IP Address Netmask Broadcast MAC Address MTU TSO MSS Enabled Type
vmk0 10 IPv4 62.75.232.6 255.255.255.128 62.75.232.127 00:26:2d:01:79:c6 1500 65535 true STATIC
vmk1 271 IPv4 10.1.1.1 255.255.255.0 10.1.1.255 00:50:56:7e:c0:9a 1500 65535 true STATIC
vmk2 655 IPv4 10.254.10.244 255.255.255.240 10.254.10.255 00:50:56:75:ef:2f 1500 65535 true STATIC
~ # esxcfg-vswitch -l
DVS Name Num Ports Used Ports Configured Ports MTU Uplinks
dvSwitch 256 10 256 1500 vmnic7,vmnic6,vmnic5,vmnic4,vmnic1,vmnic0
DVPort ID In Use Client
1 1 vmnic0
2 0
3 0
4 0
5 0
6 0
7 0
8 0
10 1 vmk0
137 0
138 1 vmnic1
139 1 vmnic4
140 1 vmnic5
141 1 vmnic6
142 1 vmnic7
271 1 vmk1
655 1 vmk2
101 0
400 0
100 0
399 0
~ # esxcfg-nas -l
Isilon is /ifs/data/PS_Customers/200107 from storage1.plusserver.de mounted
~ #
There is one special. I added a new route via: esxcfg-route -a 10.210.0.0/24 10.254.10.241 to access the NAS datastore.
But the datastore is accessable without any problems when I make a direct connection to the ESXi machine.
Any Hints?
Thanks in advance,
Stephan
Okay,
I figured something out. My shared storage (Isilon/EMC²) uses DNS Loadbalancing-technology. In other terms, the domain storage1.plusserver.de has a TTL of 0, see below:
[root@host01 ~]# dig storage1.plusserver.de
; <<>> DiG 9.3.6-P1-RedHat-9.3.6-4.P1.el5_5.3 <<>> storage1.plusserver.de
;; global options: printcmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 62571
;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 0
;; QUESTION SECTION:
;storage1.plusserver.de. IN A
;; ANSWER SECTION:
storage1.plusserver.de. 0 IN A 10.210.0.52
;; Query time: 78 msec
;; SERVER: 8.8.8.8#53(8.8.8.8)
;; WHEN: Fri Apr 15 16:33:26 2011
;; MSG SIZE rcvd: 56
[root@host01 ~]#
In other words the Forward Lookup of storage1.plusserver.de changes every second.
[root@host01 ~]# host storage1.plusserver.de
storage1.plusserver.de has address 10.210.0.58
[root@host01 ~]# host storage1.plusserver.de
storage1.plusserver.de has address 10.210.0.87
[root@host01 ~]# host storage1.plusserver.de
storage1.plusserver.de has address 10.210.0.65
[root@host01 ~]# host storage1.plusserver.de
storage1.plusserver.de has address 10.210.0.99
[root@host01 ~]# host storage1.plusserver.de
storage1.plusserver.de has address 10.210.0.59
[root@host01 ~]#
[....]
vmdx.log:
[2011-04-15 14:25:07.562 173C6B90 verbose 'SoapAdapter.HTTPService'] User agent is 'VMware-client/4.1.0'
[2011-04-15 14:25:07.563 173C6B90 verbose 'SoapAdapter.HTTPService'] HTTP Response: Client: NeedsContentLength: false UnderstandsChunking: true CanKeepAlive: true (PresetContentLength 129)
[2011-04-15 14:25:07.563 173C6B90 verbose 'SoapAdapter.HTTPService'] HTTP Response: Auto-completing at 129/129 bytes
[2011-04-15 14:25:07.563 17650B90 verbose 'SoapAdapter.HTTPService'] User agent is 'VMware-client/4.1.0'
[2011-04-15 14:25:07.563 17650B90 verbose 'SoapAdapter.HTTPService'] HTTP Response: Client: NeedsContentLength: false UnderstandsChunking: true CanKeepAlive: true (PresetContentLength -1)
[2011-04-15 14:25:07.563 FFFB7B10 info 'App' opID=8C6319D4-00000245-67] [VpxLRO] -- BEGIN task-internal-272 -- -- vpxapi.VpxaService.searchDatastore -- 52cfe477-24b8-df9f-555a-c176d73aaa37
[2011-04-15 14:25:07.618 FFFB7B10 error 'App' opID=8C6319D4-00000245-67] [VpxURL::GetLocalPath] locator 10.210.0.81 in URL not found on host: netfs://10.210.0.81//ifs/data/PS_Customers/200107/
[2011-04-15 14:25:07.619 FFFB7B10 verbose 'App' opID=8C6319D4-00000245-67] [VpxaDatastoreContext] Resolved URL netfs://10.210.0.81//ifs/data/PS_Customers/200107/ to localPath
[2011-04-15 14:25:07.661 FFFB7B10 error 'App' opID=8C6319D4-00000245-67] [VpxURL::GetLocalPath] locator 10.210.0.81 in URL not found on host: netfs://10.210.0.81//ifs/data/PS_Customers/200107/
[2011-04-15 14:25:07.671 FFFB7B10 verbose 'App' opID=8C6319D4-00000245-67] [VpxURL] Resolved localPath /var/log/vmware/vpx/ to URL netfs://62.75.232.6//var/log/vmware/vpx/
[2011-04-15 14:25:07.671 FFFB7B10 verbose 'App' opID=8C6319D4-00000245-67] [VpxURL] Resolved localPath /var/log/vmware/vpx/ to URL netfs://62.75.232.6//var/log/vmware/vpx/
[2011-04-15 14:25:07.671 FFFB7B10 verbose 'App' opID=8C6319D4-00000245-67] [VpxURL] Resolved localPath /var/log/vmware/vpx/ to URL netfs://62.75.232.6//var/log/vmware/vpx/
[2011-04-15 14:25:07.671 FFFB7B10 verbose 'App' opID=8C6319D4-00000245-67] [VpxURL] Resolved localPath /var/log/vmware/vpx/ to URL netfs://62.75.232.6//var/log/vmware/vpx/
[2011-04-15 14:25:07.672 FFFB7B10 verbose 'App' opID=8C6319D4-00000245-67] [VpxURL] Resolved localPath /var/log/vmware/vpx/ to URL netfs://62.75.232.6//var/log/vmware/vpx/
[2011-04-15 14:25:07.672 FFFB7B10 verbose 'SoapAdapter.HTTPService' opID=8C6319D4-00000245-67] HTTP Response: Complete (processed 782 bytes)
[2011-04-15 14:25:07.672 FFFB7B10 info 'App' opID=8C6319D4-00000245-67] [VpxLRO] -- FINISH task-internal-272 -- -- vpxapi.VpxaService.searchDatastore -- 52cfe477-24b8-df9f-555a-c176d73aaa37
I tried to add the datastore with an IP instead of the dns record... And voila it works!
Somebody encountered the same problem?
Iam still a bit confused... Why does it work when I make a direct connection to the ESXi hosts? First I thought i was luckily with the load-balancing, but it works EVERY time. Some ideas? For me it seems, this is a feature of storage systems which was not taken into account by VMware.
Best regards,
Stephan
Hi,
today i had a Webex Session with a VMware engineer. The summary is that VMware don't support all "Isilon Smartconnect" features. Sadly there is no hint in the HCL of VMware, but I found a paragraph in the the Isilion documentation, see attachment. I will create a feature request, and hopefully VMware will consider this feature in one of the next vSphere releases.
Best regards,
Stephan