Recently I have switched from NSX-V to NSX-T.
After deploying the Tier 0 Router and setting up the BGP neighbor of the main firewall, they do not connect.
When I look at the BGP neighbor status in the NSX-T console, it reads DOWN with this error: "There are some unknown runtime issues."
The main firewall is pfSense and the NSX-T version is 2.5
Anything that I am doing wrong or did something bug out?
This is the configuration on the Tier 0 Router:
And on the pfSense firewall (with frr):
Final update:
It now works. pfSense configuration was correct. The solution was to remove the ESXi host from the Uplink N-VDS switch it seems like.
Do we have point to point connectivity between Tier-O and pf Sense ?
Please do share respective VRF routing table and BGP neighbour summary from Tier-O and pfsense global routing table , i can see your are trying a EBGP connection , If you are unsure about BGP debug & validation , please do change the AS and make it same on both the sides(65950) for time being.
I can ping the Tier 0 IP address from the pfSense router, and I can ping from the Tier 0 Router to pfSense.
(I changed both AS numbers to 65950 but it did not change anything; do note that on the pfSense router, the State changes from Connect to Active and back every now and then)
This is what I see on the pfSense router:
IPv4 Unicast Summary:
BGP router identifier 192.168.20.1, local AS number 65950 vrf-id 0
BGP table version 6
RIB entries 9, using 1440 bytes of memory
Peers 1, using 13 KiB of memory
Neighbor V AS MsgRcvd MsgSent TblVer InQ OutQ Up/Down State/PfxRcd
192.168.20.3 4 65950 0 0 0 0 0 never Active
Total number of neighbors 1
Routing table:
Codes: K - kernel route, C - connected, S - static, R - RIP,
O - OSPF, I - IS-IS, B - BGP, E - EIGRP, N - NHRP,
T - Table, v - VNC, V - VNC-Direct, A - Babel, D - SHARP,
F - PBR,
> - selected route, * - FIB route
K>* 0.0.0.0/0 [0/0] via 94.x.x.1, em0, 00:03:12
K>* 46.4.x.x/32 [0/0] via 94.x.x.1, em0, 00:03:12
C>* 94.x.x.0/23 is directly connected, em0, 00:03:12
S 172.27.224.0/20 [1/0] via 192.168.254.146, em1, 00:03:12
K>* 172.27.224.0/20 [0/0] via 192.168.254.146, em1, 00:03:12
C>* 192.168.20.0/24 is directly connected, em2, 00:03:12
C>* 192.168.254.0/24 is directly connected, em1, 00:03:12
About showing the table of the Tier-0 router, I don't know how I can access it and run the commands on that you are asking for.
Could you explain that to me?
I have done some additional troubleshooting.
I ran tcpdump on the pfSense interface that faces the Tier 0 Router, and I see that BGP packets are being send to the Tier 0 Router, however, no packets are being received.
I can however, ping the Tier 0 router.
Do you have a gateway firewall configured on the T0?
I'm not sure what you mean.
If you mean that if I have a default gateway set on the Tier 0 Router, then yes, I do have that set to the pfSense router IP of the interface facing it. (192.168.20.1)
Update:
It seems to be an issue with the config on the pfSense side. When I set up frr on a Ubuntu Server VM and use BGP on that, it works.
Final update:
It now works. pfSense configuration was correct. The solution was to remove the ESXi host from the Uplink N-VDS switch it seems like.
Hi MasterWayZ!
Could you share a bit more information please? I've got the same issue.
What do you mean by "ESXi host from the Uplink N-VDS switch", are you not running ESXi mgmt over n-vds anymore?
Thanks in advance!
// Carl
Hi,
I meant to only have the ESXi host on the Overlay N-VDS and not on the VLAN one.. That fixed it for me.
I assume you're not running NSX-T2.5.1 then? Because my setup only has 2pnics (fully collapsed), both attached to the same n-vds, so i can't really take the hosts off any n-vds.
When I made this post I believe I was running NSX-T 2.5
What I had was I had multiple NICs on my server. One was attached to a regular port group with the normal management vmknics and some VMs on it. The other NICs I assigned to the Overlay N-VDS. I'm not sure if something important changed in the .1 release, I don't think it has. If you'd like I can deploy it in a lab and document setting it up so you can see how I did it if that helps. Or if you have any questions feel free to ask.
Okay so I've done my fair share of messing about with this and has reached a conclusion:
This error message simply means that the BGP peer isn't established, as in that the BGP isn't configured on the other side/misconfigured in any way.
On any other router software this would simply be a "state connecting", but here it throws and error to set you down the wrong path.
If you want more debug information:
There you'll see information about BGP state that'll be able to help you along your path, hopefully this error message is corrected in the future to not set people down the "fuck idk what i'm doing" path just because they're new working with NSX-T.
MasterWayZ Thanks for your assistance and support too!