http://www.routetocloud.com/2014/12/nsx-v-troubleshooting-connectivity-in-l2-segment/

[rbenhaim]

This blog post we describe the methodology to troubleshoot L2 connectivity within the same Logical switch L2 segment.

Some of the steps here can and should be done via NSX GUI,vRealize Operations Manager 6.0 and vRealize Log Insight,  so see it like education post.

There are lots of CLI commands in this post .

[TronAr]

Wow, lots of CLI there, thank you!

Now, I have some confusion with MTU

In your tests you make a ping -s 1570, but that would create an oversized frame AFAIK.

Because ICMP header of length 8, IP (20) and ethernet (14)  makes for 42 extra bytes, that takes you to 1612 which is over 1600.

I run into your post because I'm facing exactly this issue, and the UI logical switch monitor test fails as well.

(but full size VM packets do work though, because VXLAN overhead amounts to "just" 64 bytes.

Where am I missing it ?

TIA,

-Carlos

[rbenhaim]

Wow, lots of CLI there, thank you!

Now, I have some confusion with MTU

Thanks

In your tests you make a ping -s 1570, but that would create an oversized frame AFAIK.

Because ICMP header of length 8, IP (20) and ethernet (14)  makes for 42 extra bytes, that takes you to 1612 which is over 1600.

You are correct with ping ++netstack=vxlan, total overhead is 42 bytes and the total frame size is on the wire is 1612 bytes (i'm adding screenshot).

~ # ping ++netstack=vxlan 192.168.250.53 -c 1 -s 1570 -d

PING 192.168.250.53 (192.168.250.53): 1570 data bytes

1578 bytes from 192.168.250.53: icmp_seq=0 ttl=64 time=0.723 ms

capture it with the command:

pktcap-uw --uplink vmnic0 -o /var/log/1.pcap

From the capture it's look like no VXLAN encapsulation occurs with this ping ++ command.

I run into your post because I'm facing exactly this issue, and the UI logical switch monitor test fails as well.

please note: for UI logical test to successful you need VM attached to logical switch!!!

(but full size VM packets do work though, because VXLAN overhead amounts to "just" 64 bytes.

Where am I missing it ?

The VXLAN overhead is 50 bytes, read my post here:

NSX Minimum MTU | VMware Professional Services

does it answer your question ?

but

[TronAr]

Hmm, sort of.

Lets get first the 50/64 out of the way: Yes, vxlan adds 50 on top of ethernet.

Which is 64 on top of the L3 PDU, from where came my mistake.

But... I'm still confused. Which is good, because then I know I'm missing something here.

MTU as set in the ESXi interface is L2 or L3 ? Because 1500 is standard L3 MTU, and you

usually have 1518 L2 MTU. Now, it begins to confuse me where the switches also have MTU of 1500 by default,

cause I would say switches have litle business on L3, but then its their "cargo"  (max PDU).

So a ping -s 1570 generates a frame of 1612 which should fit in the 1618 bytes that teh maximum frame

should be able to carry. But it's not getting through.

Also, I did have a VM attached to the LS, two in fact, and the test failed.

(and a full size frame was delivered ok between VMs in different hosts)

I have to see what's going on. This is in fact a nested lab on top of workstation.

And I did see the LS monitor test running fine once...

[TronAr]

Oh well, it would seem that workstation does not support going over 1600 byte frames.

Two VMs directly connected to a local (host only) bridge, with vmxnet3 drivers, seem to work upto that.

(MTU 1584)

I've seen some posts of people claiming it works under workstation, but I guess they did not actually tested the

LS monitor ping thingy. The whole ipstack thingy is not that much mentioned either. It would be nice calling it VRF

Still, I managed to see the ping "green" at some point, so tere might be something else lurking here.

-Carlos