NSX Edge Nodes not communicating

[pm-walid]

Hi,

We have deployed two NSX Edge Nodes to form a cluster, we have a two DVS on our environment, one is used for management/vmotion.. and the other is for production.

We have 5 ESXi servers all configured as transport, we have configured application segments and it's working properly. Now we wanted to add external access to our environment, so we deployed the NSX Edge Nodes and configured Host TEP overlay (VLAN 0), and also Edge TEP overlay (VLAN 70). 

Now both the edge VMs cannot communicate to the external network and also ping between them, note that the ports on the physical switch are configured with Access VLAN 70. And communication of the Host TEP is working properly.

Did we miss a step ? Is BGP configuration mandatory ? 

We followed exactly this step by step by VMware, apart from using VLAN 0 for Host TEP :

https://docs.vmware.com/en/VMware-NSX-T-Data-Center/3.2/quick_start/GUID-E6E40D82-DB4D-4D36-8912-00A...

Thank you

[Sreec]

First, you need to fix TEP connectivity issues. Any specific reason why you are using VLAN 0 ? I would recommend using a common VLAN for Edge and Host and please ensure MTU is set correctly. 

https://kb.vmware.com/s/article/83743 

[pm-walid]

Hi,

Thank you for your feedback, we are using vlan 0 since we don’t want to use vlans (is it mandatory ?). Is also the BGP configuration mandantory, as you see in the official documentation it doesn’t mention anything about BGP.

thank you

[Sreec]

An untagged VLAN is not a great choice from a design perspective, which is why I recommended sticking with a proper VLAN.  BGP is not mandatory. TEP must work irrespective of the routing protocol you are using in Edge Uplink interfaces

[pm-walid]

The Host TEP which is currently using VLAN 0 is working properly (communication between esxi and VMs work as designed), but for the Edge TEP using VLAN 70 it doesn't work. As per the Vmware documentation we don't require to create a dedicated port group for edges (As I have seen in many tutorials), the physical port is in Access 70.

To troubleshoot, we created two VMs and assigned them IPs similar to the edges and VLAN 70 and they can communicate. So it's definitely something to do with the edge configuration which does not allow external communication.

Any idea on what can be causing the issue ?

Attached are the screenshot for deploying the Edge.

Thank you

[Sreec]

Kindly share your uplink profile(uplinkprofile2)  and Edge Uplink portgroup VLAN configuration screenshots. 

[pm-walid]

Hi,

We didn't create an Edge Uplink portgroup VLAN, we used the PG-ALL-VLAN port group, that's the source of my confusion, should we create a dedicated port group for Edge and put VLAN 70 in the port group configuration ?

Thank you

[jchilton]

Hi, not a network engineer but think I’ve got my head around a similar problem. 

if your edge is a VMware appliance and it’s hosted within your collapsed cluster ie not on a seperate host within its own transport zone, you can’t connect the Edge nvds switch to vDS Port Group. It has to be to an NSX vlan segment that you need to create (as an all vlan e.g 1-4094). Apparently the host that has the Edge VM can’t share the VTEP traffic within itself. 

I found this useful https://fojta.wordpress.com/2020/11/12/nsx-t-3-1-sharing-transport-vlan-between-hosts-and-edge-nodes...

i think the VMware documentation around the edge especially for small environments is shockingly poor. I’m trying to use the latest version v4.1 but majority of docs/blogs are centered in NSX-T. I still can’t get my head around the Edge fast path interfaces and whether to use more than 1 nvds. Is fp-0 always the vtep interface??

The Quick Start guide is also incorrect or aimed at an architecture with seperate hosts/Edge as these use a vDS port group also. V confusing.

https://docs.vmware.com/en/VMware-NSX/4.1/quick_start/GUID-78489E7A-1F6F-4317-BD8B-DDF59FEF9860.html

Hope that helps.