Six physical switches, is this best practice?

[HendersonD]

You can see how we have our vSwitches configured below. Is this layout optimal or should we consider changes?

[Troy_Clavell]

we tend to have two pNICS for our COS and vMotion and they go on vSwitch0, each is standy for the other. Also, if you can, load balance the NICs with an onboard and PCI NIC. I don't think vmotion needs two dedicated NICs. I would give more NICs the my virtual machine port group.

[patrickds]

It's considered best practice to separate your iSCSI traffic from the rest.

Using 6 nics i would have put Service Console and Vmotion on vswitch0 (using either loadbalancing on portID for both, or specific failover to put SC on vmnic0 and vmotion on vmnic1 with the other as standby)

vswitch1 has a Service Console + Vmkernel for iSCSI, on a different network (either physically separated using different switches, or using VLANs) + the VM iSCSI for VMs directly accessing the storage (like you're doing already)

vswitch2 only for VM Network.

You'd also better check, if you're using dual port pNICs, that you don't use both ports from a single card on 1 vswitch, to provide redundancy when an adapter fails.

[HendersonD]

I should have mentioned that my three ESX servers are blades in an IBM BladeCenter. The BladeCenter has 6 physical switches installed in it and each switch has 4 external ports. We use two connections from each switch bonded (etherchannel) to provide a 2gig connection between the switch and my Cisco 6509 core switch. For redundancy, each pair of physical nics (for example vmnic0 and vmnic1) are plugged into two different blades on my core switch just in case I lose a blade. Separate vlan are setup on my core switch to keep traffice separate:

vlan 125 for my VM Network which is routed

vlan 126 for iSCSI traffic, not routed

vlan 127 for vMotion traffic, not routed

With this setup I do not think that network performance is an issue. I do have my Service Console in with my VM Network. I was wondering if the Sertice Console should be in with my VMotion network instead?

[Troy_Clavell]

With this setup I do not think that network performance is an issue. I do have my Service Console in with my VM Network. I was wondering if the Sertice Console should be in with my VMotion network instead?

Service Console and Vmotion I typically see together on vSwitch0. As I posted that is how our 70+ ESX hosts are setup, which includes both rack mount and blades.

[HendersonD]

Two last questions:

Is there a disadvantage to having the Service Console on the VM Network?

If I wanted to move the Service Console to the VMotion network would I just create a Service Console in the VMotion vSwitch and delete it out of the VM Network vSwitch?

[Troy_Clavell · Accepted solution]

Is there a disadvantage to having the Service Console on the VM Network?

Security. Best practice is the keep your service console on a private segment.

If I wanted to move the Service Console to the VMotion network would I just create a Service Console in the VMotion vSwitch and delete it out of the VM

anytime you make changes to the service console networking you will probably lose connectivity. My suggestion would be to migrate all VM's off the host, make your network changes.

Here's a good KB that may be somewhat handy.

http://kb.vmware.com/kb/4309499

[patrickds]

>>

vlan 125 for my VM Network which is routed

vlan 126 for iSCSI traffic, not routed

vlan 127 for vMotion traffic, not routed

Is this ESXi?

I see you don't have a SC on your iSCSI network, which would be required for ESX (not for ESXi).

That's why i assumed you were using the same network for iSCSI as for the rest (and using the SC on vswitch0 for iSCSI as well), I'm not really accustomed to seeing ESXi installations yet.

Anyway, having 2 nics dedicated for vmotion is overkill. I'd put SC and Vmotion on 1 vswitch, iSCSI on a second, and VMs on a third.

You could even combine SC, Vmotion and VMs on 1 vswitch with 4 physical nics, but that would make the configuration less transparent.

[HendersonD]

This is not ESXi, it is ESX 3.5 fully patched. Why do I need a Service Console on my iSCSI network?

[mcowger]

The CHAP initiator runs in the service console space.

--Matt

VCP, vExpert, Unix Geek

[HendersonD]

This is all starting to make a bit more sense now. I found this quote online:

"The only instance in which it would be REQUIRED to add a Service Console port group to the same vSwitch as the iSCSI VMkernel port group would be when the iSCSI network is physically separate. Thus, the only way to provide connectivity from the Service Console to the iSCSI target would be to add a Service Console port group on the same network. Otherwise, as long as the Service Console has IP connectivity to the iSCSI target, then it will work."

So I started wondering why my setup works fine without a Serivce Console on my iSCSI network. I dug a little deeper and found this is what is configured for networking on the current Service Console port which is on my VM Network:

IP Address: 10.121.125.108

Subnet Mask: 255.255.252.0

Gateway: 10.121.125.1

The problem is the subnet mask. With this mask, the IP address range is from 10.121.124.1 - 10.121.127.254. My iSCSI network is in vlan 126. With this mask the Service Console has IP connectivity to the iSCSI target. I think I should add a SC to the iSCSI network and change the mask on the VM Network to 255.255.255.0 so the range is 10.121.125.1 - 10.121.125.254. This will allow the router that connects these networks (my 6509 core switch) to do its job. Right now it appears I have traffic from my VM Network and my iSCSI network intermingled which I am sure degrades performance. I looked at the subnet mask on my iSCSI network and it to is set to 255.255.252.0, again it appears it should be changed. Once I make these changes then I need to get the Service Console off my VM Network and onto my VMotion network which seems to be best practice.

Does all of this sound like I am heading in the right direction?

[patrickds]

Sounds like a perfectly good idea.

Just one thing: your SC is not actually on your VM Network, it's just sharing a vswitch and its physical connections.

You can move the SC onto the other vswitch, which also has the vmotion port group, and keep them both in their own VLAN by setting up VLAN tagging on the port groups and trunking on the physical switches to allow both VLANs tagged packets to enter the ports.

[HendersonD]

Can I just

- Add an SC to vSwitch2 which is used for vMotion

- Add another SC to vSwitch1 which is used for iSCSI

- Delete the SC from vSwitch0 which is for my VM Network?

I have vSwitch2 (used for vMotion) attached to two physical switches with etherchannel setup to my core switch giving it a 2gig connection. It is also on its own non-routable vlan 127

I have vSwitch1 (use for iSCSI) attached to two phyiscal switches with etherchanne setup to my core switch giving it a 2gig connection. It is also on its own non-routable vlan 126

If I add a SC to vSwitch1(vMotion) and vSwitch2(iSCSI) and delete it off vSwitch0(VM Network), do I need to put in gateways for the SC in order to make it routable and maintain connectivity or can I leave out the gateway thus leaving vlan126 (iSCSI) and vlan 127(vMotion) non-routable?