Cisco Nexus 1000v VMotion

[eljow]

Hi,

I wonder if you could help me out though, I am struggling with Cisco Nexus 1000v.

I am trying to figure out how to do vmotion on the Cisco Nexus 1000v. At the moment I have both cold/hot migrations taking place but I am not sure if its best practise the way I have configured it.

I have created a vmotion port-profile that is used by the vmk interfaces, which is as follows,

port-profile type vethernet VMotion

vmware port-group

switchport mode access

switchport access vlan 50

no shutdown

system vlan 50

state enabled

Now if I add vlan 50 as a system vlan, the vmotion traffic will use the main system-uplink trunk, which the control, packet VLANS go over and it will work perfectly.

Problem is I want to be able separate the vmotion traffic using a different vmnic. So I created another uplink, configuration is below,

port-profile type ethernet VMOTION-LINK

vmware port-group

capability uplink

switchport mode trunk

switchport trunk allowed vlan 50

no shutdown

system vlan 50

state enabled

for that port-profile uplink I have added a different VMNIC and allowed the relevant configuration on the upstream switch. So now my vmk interface traffic for vmotion will now use the VMOTION-LINK, I have also remove the system vlan 50 from the system-uplink port-profile but it does not work and the migration bombs out at 10%. I have also tried creating the VMOTION-LINK as an access port aswell, which is as follows,

port-profile type ethernet VMOTION-TEST

vmware port-group

capability uplink

switchport mode access

switchport access vlan 50

no shutdown

system vlan 50

state enabled

again changing the upstream switch to an access port from a trunk port but it does not work and bombs at 10%. I can only get it working on the system-uplink port-profile.

My question is this, is what I am trying to achieve even possible with the Nexus 1000v and separating the Vmotion traffic onto a different VMNIC from the main system port-profile?

Any help would be great.

[mmehta76]

Hi,

this is very much possible with Nexus 1000V to separate VMotion traffic on a VMNIC from the other NICs used for data VLANs. I am suspecting that you may have the VLAN 50 configured on both data UPlink and the VMotion UPlink. We do not allow system VLANs to be deleted from a live port-profile since they are carrying critical traffic. In order to remove it from Data UPlink and easy way will be create another uplink port profile with all data VLANs except system VLAN 50, and then move the data PNICs to this new one in the VCenter.

Let me know if it works. could you please provide information on the software version you are using, as well as configuration of all the uplink port-profiles.

Munish.

[eljow]

Hi mate,

I definatley don't have the VLAN tagged with any other profile. I learned the hard way trying to remove a system vlan for a live port-profile, it took me ages to figure out how to remove it. In any case i don't think that is the problem.

As for port-profile configuration, you will find this information below,

Main System-Uplink

port-profile type ethernet system-uplink

vmware port-group

capability uplink

switchport mode trunk

switchport trunk allowed vlan 100 -102

no shutdown

system vlan 100 - 102

state enabled

Data Uplink Port Profile

port-profile type ethernet data-uplink

vmware port-group

capability uplink

switchport mode trunk

switchport trunk allowed vlan 150 - 155

no shutdown

state enabled

Above are the main uplinks, system uplink and data uplink that carries vm traffic. The VMotion i have tried three sets of configuration, firstly, if i configure the VMotion with the following below and assign an extra system vlan (VLAN 50) to the system-uplink and assign the below port-profle to the vmk interfaces, it will work perfectly but i don't want this i want it seperate from the system-uplink.

Used by VMK Interfaces.

port-profile type vethernet VMotion

vmware port-group

switchport mode access

switchport access vlan 50

no shutdown

system vlan 50

state enabled

Secondly, if i remove the system vlan 50 from the system-uplink and configure a new uplink with a seperate VMNIC, as in the below,

port-profile type ethernet VMOTION-LINK

vmware port-group

capability uplink

switchport mode trunk

switchport trunk allowed vlan 50

no shutdown

system vlan 50

state enabled

and assign the VMK interfaces the Vmotion VMK port-profile it won't work using the new VMOTION-LINK. Thirdly and just for a test i changed the VMOTION-LINK, so its a access port uplink instead of a trunk port as in the below,

port-profile type ethernet VMOTION-TEST

vmware port-group

capability uplink

switchport mode access

switchport access vlan 50

no shutdown

system vlan 50

state enabled

it still does not work correctly.

I have checked that the system vlan 50 is not part of any other uplink ports and is only configured on the VMOTION-TEST

or VMOTION-LINK depending what i have configured at the time. The upstream switch matches all port-profiles, so i cannot not understand why it will only work using the system-uplink profile. Does the VMotion traffic have to pass down the same uplink as control and packet, i would doubt it but its the only way i can get it working?

cheers

[mmehta76]

Hi,

thanks for sharing the profile configurations. Nothing obvious stands out. Can you please provide additional information as below:

1) Which software version are you using?

2) From ESX host: "vemcmd show port", "vemcmd show trunk". "vemcmd show profile". Ideally "vem-support all" command captures all the needed info in a file.

3) Configuration of corresponding ports on the upstream switch.

4) Since you are vmotioning from one host to other, please provide the above on both hosts.

Munish.

[DOakman]

We ran into a similiar issue today with a similiar configuration. Vmotion was failing when we had vmotion on a dedicated Portgroup (isolated Layer 2 Vlan upstream) and Management on a seperate port group (routeable L3 subnet upstream).

We ESXi hosts could ping each other on both the Management and Vmotion subnets.

Vcenter could access hosts through management IP, but not vmotion (due to L2 isolation).

We moved changed the vmkernel config to have vmotion and management on the management subnet and vmotion worked fine.

ESXi version 4.1

Nexus 1000v version 4.0(4)SV1(3a)

We want to go back and move this back to dedicated vlan, but need to figure out why it is not working.

We are having 1 other problem after vmotion occurs in that Vms on a particular port group loose network access. We are only able to get the vm back up ad running after changing port groups and then changing back to original.

DaveO