vMotion between two hosts doesn't work

[ZenWare]

I have two hosts which I have connected directly via 10G NICs bypassing a switch.

I want to use this connection for vmotion and have followed all the instructions.

I created a new vSwitch on each host, with a vMotion VMKernel port group, and the vmnics connected to the vSwitch.

The IP addresses are for dedicated subnet seperate from the management and vmnetwork.

Since vCenter forces you to use a gateway, I checked "override Gateway" and tried a common gateway address on the new subnet that didn't exist.

As this didn't work, I then used the other hosts as gateways, different gateway addresses on each vSwitch.

vMotion is enabled only on this switch. However.... It doesn't work!

Since vMotion doesn't use the designated route, I tried SSH into both the hosts to ping the other host. No ping!!

I tried pinging the host's own IP (for vmotion vmkernel) and NOTHING!

Can someone help me, I think the problem is either:

1. Need a crossover cable (the NIC supports crossover but vmware might not?)

2. Need a distributed switch instead of a basic vswitch?

2. Need a gateway address?

THANKS

[RajeevVCP4]

Just test , enable vmotion on vmkernel port then try to vmotion.

If it is work then problem in your network not from vmware side

[ZenWare]

Hi Rajeev, thanks for your reply.

The vmkernel adapter is using the vMotion TCP/IP stack. The only enabled service is vmotion.

Thanks

[NathanosBlightc]

Answer to Q1: No its not related to crossover / straight cables! Don't worry about VMware support

Answer to Q2: You can do it with the Standard vSwitches. It doesn't need to be in a VDS! different between VSS and VDS are related about their Management Plane and some advance networking feature, not basic operation of networking.

Answer to Q3: ‌Because you probably set two IP addresses from a single IP subnet, So no need to set default gateway for the VMKernel port of your hosts.

But after all check the following:

1. Configured Subnet (/x) for the vmkernel ports must be same, check it again.

2. Try with another cable if you can, or check this one between two different systems to ensure about its healthy

3. Investigate there is NOT checked the vMotion checkbox in the default (first) switch if you assign another vSwitch / Portgroup for the vMotion operation and it checked vMotion checkbox anyway.

4. Check the dedicated uplink (vmnic) for this new vSwitch.

5. Do you have network connectivity between that old vSwitches assigned for the host management or there is also no network sense ?!

[depping]

How did you test the ping? Did you select the correct vmkernel interface? Details on how to do that you can find here: VMware Knowledge Base

[ZenWare]

This is odd.

There are 2 Interfaces: vmk0 (management, vmnetwork) and vmk1 (vMotion stack).

I ran "vmkping -I vmk1 10.0.0.92" and resulted in

Unknown interface 'vmk1': Invalid argument

I ran the same command for vmk0 (to a different IP on the subnet) and it PINGS properly.

Why would vmk1 be unkown?

[ZenWare]

THANKS AMIN!!!

1. Yes, the subnet is the same. it's 10.0.0.0/24 for both hosts.

2. The cables are fine; replaced them and tried migrating a VM again. Doesn't work.

3. No other vMotion in other switches. Do I need a seperate portgroup? I thought that was only for VM networking and not vMotion. My vSwitch only has a vmKernel portgroup for the uplink port (vmnic) and no portgroup for VMs.

4. Checked the dedicated uplink. The settings are fine. The gateway address is 10.0.0.254 (which isn't an existing port) as I can not leave this empty.

5. The network connectivity exists for the management link; this is how vmotion is happening. Through this vswitch.

[mguidini]

Why are you using vMotion TCP IP Stack? Hosts are connected via a single Layer 2 switch isn't?

Use the Default Stack, enable vmkernels for vMotion, assign IPs to the vmkernels and this option is not going to ask you for a gateway. vMotion TCP IP Stack is only used when you need to route the vMotion network (L3). Doesn't seem like your case here.

1. Need a crossover cable (the NIC supports crossover but vmware might not?) (No)

2. Need a distributed switch instead of a basic vswitch? (No)

2. Need a gateway address? (No)

Regards,

[depping]

Indeed, you don't need to use the TCP/IP stack for vMotion. it is a cross-over cable so it will be L2 and not L3. I would go back to basics first, use a single vSwitch, create a basic VMkernel interface with IPs in the same subnet on both sides. Keep it simple.

[ZenWare]

Done. Back to basics.....Still nothing.

[mguidini]

List the vmkernels:

esxcfg-vmknic -l

Ping results to the destination host?

vmkping -l vmk<X> <destination ip>

Ping results with packet size to check jumbo frames?

vmkping -l vmk<X> -s 1472 <destination ip>

vmkping -l vmk<X> -s 8972 <destination ip>

Delete the vMotion vmkernel and re-create it. There might be a misconfiguration somewhere

[NathanosBlightc]

ZenWare wrote:

THANKS AMIN!!!

1. Yes, the subnet is the same. it's 10.0.0.0/24 for both hosts.

2. The cables are fine; replaced them and tried migrating a VM again. Doesn't work.

3. No other vMotion in other switches. Do I need a seperate portgroup? I thought that was only for VM networking and not vMotion. My vSwitch only has a vmKernel portgroup for the uplink port (vmnic) and no portgroup for VMs.

4. Checked the dedicated uplink. The settings are fine. The gateway address is 10.0.0.254 (which isn't an existing port) as I can not leave this empty.

5. The network connectivity exists for the management link; this is how vmotion is happening. Through this vswitch.

[NathanosBlightc]

as I can not leave this empty.

You can set 0.0.0.0 anyway if you don't want to use any default gateway IP address.

Did you ever check these log files in the time duration when you run vMotion operation?

/var/log/vmkernel.log

/var/log/vpxa.log

[ZenWare]

mguidini.

I deleted the vmotion vmkernel along with the vswitch.

I created a regular TCP/IP stack with vmotion enabled instead of creating a vmotion stack.

PING works between hosts using "vmkping -l vmk<X> <destination ip>" on this network.

However, vmotion is still not conducted on between the designated vmnics. It still uses the management network!

[daphnissov]

Ensure you do NOT have vMotion enabled on your management vmkernel port and that you DO have vMotion enabled on your dedicated vMotion vmkernel ports.

[ZenWare]

This is very very odd.

I was migrating only VM's that were turned off. (We are currently rebuilding our test environment and don't have all the servers on until its ready)

I just tried migrating a VM that was turned on and it used the 10G vmnic for traffic!!

I guess vmotion applies only to the migration of live VMs and not turned off VM's?

Is this normal? So basically need to turn on all VM's in order to use the vmotion stack.

[daphnissov]

Yes, a vMotion is defined as a live migration (i.e., a running VM) between systems. If the VM is powered off this is just a reregistration or a cold migration. My book (second link in signature) will tell you more.

[cloudsuresh]

To communicate one device with another we use cables and switches.

If you do not wish to use switch/hub then you choose cross cable. 

"Host A to Host B without switch use cross cable.

IP address of host A and host B should be in the same network. 

Example: Host A 192.168.1.10

Host B 192.168.1.20

You established physical connection and configured ip address.

Now you ping from Host A to Host B.

After physical