vSAN stretch cluster witness traffic separation

[RafalSte]

I understand the concept of having different subnets with individual static routes to witness on both sites.

Examples I have seen so far don't mention doing the same with traffic to witness management vmk0.

Should I create additional vmkernels on each esxi host with static routes exactly the same way?

Another issue that might be of concern: I enabled Jumbo Frames on management network, and witness site id behind L3 route where MTU is smaller - does it require creating another mgmt interface on each esxi with smaller mtu?

One more question - when I add esxi witness host to vCenter - should I use ip of vmk0 of witness?

[CyberNils]

Using witness traffic separation with vmk0 should not require static routes since you can use the default gateway of the Default TCP/IP stack.

I would recommend using 1500 bytes MTU on the management network to avoid issues with this network in case you make a mistake.

When you add esxi witness host to vCenter, I would recommend using the FQDN of the witness, but IP of vmk0 should work as well.

[RafalSte]

I have some doubt about it:

default gateway is on Site-A.

So if I use that and site a fails than site-b won't reach witness which is why we want static routes.

As for jumbo frames, I have backup solution moving data directly from esxi hosts to deduplication device so I would wan't to keep it.

So maybe I should tag mgmt traffic alongside vsan witness traffic on separate additional vmk with distinct routes on both sides?

And then tag witness traffic on witness vmk0 and do not use second interface on witness?

[CyberNils]

We usually have different management networks on each site so that they don't share a common gw.

[RafalSte]

Ok, but vCenter has IP on both of them?

I ran into another issue - since I used witness traffic separation and added additional vmk to esx hosts with different subnets on both sites I later configured two isolation addresses according to VxRail docs I was following:

das.isolationaddress0 IP address of the vSAN default gateway of preferred site.
das.isolationaddress1 IP address of the vSAN default gateway of secondary site.

Now I have warning on each of the hosts: vSphere HA agent on this host could not reach isolation address: 

They show that it cant reach ip of opposite site gateway which is expected in this configuration.

So what I misunderstood about it?

[CyberNils]

No, vCenter has a single IP on a network which is stretched between both sites so that it can fail over between them. This may be one of the ESXi management networks or a different one.

Did you also set das.usedefaultisolationaddress = false?

[stefanek81]

Hi,

I have usedefaultisolationaddress set to false. Since last post I also permitted traffic crossing sites from esx hosts to isolation addresses so both isolation addresses are pingable from all hosts unless interlink between sites fail.

vCenter has separate port group on VDS. However it is on same L2 vlan as management vlan that spans across sites and hosts from both sites share subnet and gateway.

L2 intersite is connected on 100Gbps link (2x100Gbps coming soon), but L3 layer comes over firewall on 1Gbps, thats why I put all hosts mgmt in same L2 domain.