ESXi 4 and NetApp NFS

[vChr1s]

Hey all. So I have a couple of things to ask here. First, I'm having a strange problem mounting NFS volumes.

I have an ESXi host with IP addresses .100 and .101. ESXi was installed with the management nic(s) using .100. I added a new vSwitch w/ vmKernel port .101 for IP storage. I have a NetApp FAS2020 for storage and have created the volume with the appropriate security. In fact the FAS has a wizard to create volumes for VMware.

Now I've built ESX environments before and I've always used roughly the same config for the NICs. 2 for Service Console/MGMT, 2 for IP Storage, 2 for vMotion, and 2 for Public Network. I know ESXi eliminates the service console but there is a Management Network by default (I have a question about that later).

My problem is, when I create a volume and set security for my storage IP (.101), I can't mount anything. However, if I change the security to allow the default Management Network (.100) the storage mounts no problem.

I guess my first question is, is this normal and just something I'm unaware of in ESXi?

Secondly, I know best practice in ESX is to keep the Service Console seperate of everything else. If I should be able to mount storage with my different storage IP (.101), is the same true of the default Management Network in ESXi? Should it be kept seperate of storage and vMotion?

Thanks for your help in advance and if you need any additional info please ask.

[ITN-Tech]

Essentially, there is no way in ESXi to tell the hosts to use an “NFS” source IP if the NFS target is on the same IP space as the “Management” interface. This is a change in behavior from ESX classic, where you can specify NFS on a specific vmk kernel port interface. ESXi will use the “Management” interface, because that can route to the NFS server. The additional “NFS” dedicated interface will sit there idly and do nothing. The good news is we get an IP back. The bad news is that we have to reconfig the filers to accommodate what was the service console IP, since NFS requests will be coming from there. If you have a separate storage network (which you should) this will be a total no-op for you. Unfortunately, we also use a single subnet for Service Console (aka Management Network in ESXi) and NFS traffic.

Two simple choices:

1> Make your storage on another network. Configure the NFS specific vmk kernel port interface to use this network.

OR

2> Configure your filers to allow traffic from the Managment Network IP and remove the NFS specific vmk kernel port interface and IP.

-Gary Kuyat

[vChr1s]

Thanks for the response. I went with option 1. The issue I was having was the default port group on vSwitch0 was called "Management" something, I can't remember now. So I thought, ok, I'll keep that for management. So I created a new vSwitch with a vmkernel, gave it an IP that had access to my filer, and I couldn't connect. Yet when I changed the filer access to allow the management IP, it worked.

After some testing, I have found that it isn't necessarily the vmk, but the vSwitch. I created a new port group in the default vSwitch0 with a different IP, gave it access on the filer, and whala, it connected. I created a new vSwitch, vSwitch1, gave the vmk an IP, and gave it access to the filer, nothing...

So the steps I took, were to create a new vSwitch and vmk port group and call it "MGMT". Then reconnect the vsphere client to it. Then change around the default vSwitch and vmk to configure it for the storage network (yes, my NFS is on seperate vlan). After fixing a few vlan issues I was having, everything started working.

I just don't get why I cant create a new vSwitch & vmk and connect it to my NFS. Why must I use vSwitch0?

[ITN-Tech]

I have created exactly the scenario you describe - vSwitch0 with both the Management Network (vmk0) and an NFS VMkernel Port (vmk1) and the NFS traffic still went through vmk0. When you got it to work through an alternate switch, is it possible that you got the NFS traffic on a lower numbered port than the Management Network? It's a bit kludgy, but this would work.

To test this, I could:

1> Have my original Manaegment Network Kernel Port (vmk0)

2> Make an additional Managment Network Kernel Port (vmk1) on the same or a new vSwitch

3> Uncheck the "Managment traffic" checkbox on the Properties Dialog for vmk0 and Change the Network Label field to "NFS".

4> Change the Network Label field to "Manaegment Network" for vmk1.

NFS now works in ESXi seemingly like in ESX classic.

I don't recommend this becuase I think this solution only hides the fact that ESXi and ESX classic work differently. It's better to make your storage network a different subnet or to face the fact that the traffic is co-mingled with the Management Traffic.

[vChr1s]

Yes, the four steps you listed is essentially what I did. While my storage network is a seperate network & vlan, I decided to go this route because I have port trunking on the switch side and I wanted seperate uplinks (i.e. physical nics) for each function.

In my environment, each host has 8 pNICs. I have also migrated to dvSwitches as well. There are 4 dv Switches: MGMT, IP STORAGE, vMOTION, & VMNET. Each dvSwitch has two uplinks (pNICs). Each pair of pNICs is trunked on the switch side and configured for active/active and Load balancing is "based on IP hash" in ESXi. I know I don't get all the advantages since there can only be one target IP but I set aliases up on the NetApp so I can use the alias IPs for other things and gain some form of load balancing.

This has been working pretty well for the last week and I'm pretty satisfied. Any other suggestions you would make? Thanks

[ITN-Tech]

Without configuring ports on your NetApp to use IP in an alternate address space, your solution is the best I can think of. The key is that ESXi is sending the traffic over the lowest numbered vmk port that can meet the need, based on the target subnet.

[Josh26]

Yes, the four steps you listed is essentially what I did. While my storage network is a seperate network & vlan, I decided to go this route because I have port trunking on the switch side and I wanted seperate uplinks (i.e. physical nics) for each function.

This has been working pretty well for the last week and I'm pretty satisfied. Any other suggestions you would make? Thanks

If I read your initial post, then the above post correctly, you haven't implemented vlans correctly. This is just my assumption as you appear to only mention the last octet of your IP ranges. If your storage network is on a separate VLAN, however, with the same network address, how should a packet ever know which vlan it's mean to go across? The answer is, "whatever kludgey rule VMWare uses" and that's not the way to go about it.

Place your storage NIC on a totally different network IP range (along with your filer) and then VMWare will always know that the filer is attached to only that NIC.

[ITN-Tech]

Amen.

[vChr1s]

Your correct, I apologize. I had made some configurational changes in between posts that I forgot/didn't have time to mention. VLAN 90 is the default vlan and the vlan I am in. I am in a 10.90.10.x network. Vlan 101, the storage vlan, is in the 10.90.96.x network.

That being said, each pair of pNICs are trunked at the switch (all HP Procurve). HP vlans are either tagged or untagged. From the storage side, it looks like this:

filer a (VMware & NFS Traffic)

e0a e0b

\ /

vif0a

____|____

vif0a-90 vif0a-101

____|____

Port 23 Port 24

________

|

Multi mode Trunk on the switch -

---

> vlan 90 TAGGED (management)

-

---

> vlan 101 TAGGED (storage)

-

---

> All other vlans NO

From the ESXi side, the two pNICs for storage (dv switch IP Storage) are plugged into two trunked ports which are configured as UNTAGGED for vlan 101. The other nics are in paired trunks and configured as UNTAGGED for vlan 90.

As a test, I also get the same results when I change the vlans for the esxi storage nics trunk group to be configured as TAGGED for 90 and 101 and then configure the vlan ID in the port group for 101.

[ITN-Tech]

/24 Networks, right?

[vChr1s]

No /20. I just used what I was told. I'm assuming it's bad practice

since I won't be needing all 4096 hosts. I could probably get away

with 32 hosts. Def with 64. /26 or /27.

On Jun 2, 2010, at 12:33 PM, "ITN-Tech" <communities-