New VI3 Infrastructure Setup - Need Tips on Network Design

[AngelC2]

I'm new to VMware, just finished up VI3 training and I'm starting my first deployment so I have questions regarding best practices or even better examples of how to setup networking for my environment.

I'm hoping someone has either a similar environment or enough experience to give me some examples on how I can set it up, so I can then evaluate different solutions and choose the right one for me.

My environment will consist of 6 production ESX 3.5 Enterprise host servers, 2 test/dev ESX 3.5 Standard host servers, one VirtualCenter server. I'm running on HP hardware and using Fibre Channel SAN for storage of VM's and for now I just need to get the Test/Dev environment setup and I'm using one ESX server for that.

My network is a flat one consisting of 2 subnets...production is split up between trusted servers (10.x.x.x) and DMZ (192.168.x.x). I have my service console and VMKernel port (for SAN & Vmotion) configured on the same default vSwitch0 and I'm anticipating putting in a seperate switch for VM port groups but that's where I'm not 100% sure. I'm thinking I will need another switch for DMZ servers, etc. and will have a 2nd service console on it as well for redundancy.

Anyway, I hope that's enough info to get some good feedback on possible examples/scenarios of how I could configure my networking piece of my VI3 Infrastructure...thanks!

[ablej]

How many pNICS does each server have?

David Strebel

www.holy-vm.com

[azn2kew]

It would be helpful to know how many NICs do you have to plan your networking setup. But generally speaking, at a minimum you have 4 pNICs which will conver the following combinations.

pNIC1->Service Console/VMotion

pNIC2->VMotion/Service Console

pNIC3-4->Virtual Machine Network

pNIC5-6->DMZ Network

That would be prefer production with 6 pNICs in place with DMZ in mind. You can configure VLAN and secure it with standard best practices on www.vmware.com ->Resources it has all the details how to implement networking in ESX.

If you found this information useful, please consider awarding points for "Correct" or "Helpful". Thanks!!!

Regards,

Stefan Nguyen

iGeek Systems Inc.

VMware, Citrix, Microsoft Consultant

[altonius]

I agree with azn2kew but with one further suggestion (again depending on the count of pNICs in your servers)

That is to try and split your traffic across the onboard NICs and your PCI NICs

In our Instance we have

Onboard

pNIC0 -> Service Console / Vmotion

pNIC1 -> VM Network(s)

PCI Adapters

pNIC2 -> Vmotion / Service Console

pNIC3 -> VM Network(s)

I can't remember the last time that I actually had a failed NIC on a Motherboard or infact a failed NIC at all, but it can help with your redundancy.

[malaysiavm]

by the way, you will need to do some planning about the number of VM you plan to run on 1 host. 10:1 ot 20:1 as an example. This will impact to the total bandwidth you need to plan backend for your network connection. You will not only teaming for 2 physical NIC to support 20 VM if you plan to consolidate 20:1. Of course, from networking perspective, you may need to plan on the switch level, for redundancy and bandwidth connection within the Data center.

Malaysia VMware Communities -

[AngelC2]

Ok, I'm sorry for leaving out that big piece of info and thanks for all of those that have replied. Ok, this is my setup for each ESX server that I have:

On-board NIC = 2

PCI NICs = 4

Total = 6

From what I gather from the replies...it looks like having the same NIC for Service Console & Vmotion and on the same default vSwitch0 is ok which is what I've done since they reside on the same subnet.

Now, regarding the rest of the pNICs...I see that for redundancy of the Service Console that you recommended putting this on a different NIC on the same ESX server but this one is being used for non-Vmotion...only Standard version will be used for DEV/TEST environment. I was planning on setting up the 2nd Service Console on the DMZ side once I start setting up the TRUST/DMZ environment.

So, I would currently have for this DEV/TEST environment:

pNIC1 = Service Console

pNIC2 = SAN access

pNIC3 thru 6 = VM Network

What would be really helpful to know is...would this design be best done with the one default vSwitch0 since all of this traffic will be communicating on the same subnet of 10.x? or should I create a seperate switch for some of this? I'm thinking no, but I would want your opinions :smileygrin:

Also, for a big picture of my environment...this is what the end result is going to be:

6 ESX 3.5 Enterprise servers...3 on the TRUST 10.x network and 3 on the DMZ 192.x network. These will be using Vmotion, DRS, HA, etc.

2 ESX 3.5 Standard servers will be used for DEV/TEST...one server for DEV on TRUST side 10.x and one TEST on DMZ side 192.x >>> This is what I'm working on setting up now.

[vmwareluverz]

There is a vmware guide how to setup networking you can check out www.vmware.com click on resource.

[AngelC2]

Ok, what I got so far is vSwitch0 (default) running Service Console and iSCSI FC SAN access (no Vmotion enabled) on the same vmnic0 since they are both on the same subnet.

I then created a new switch (vSwitch1) for my VM Port Group and assigned it to a different physical NIC. The other 4 NIC's will also be added as needed to this switch for when more VM's end up on it..again...only if needed based on traffic performance, etc.

By the way, I have no clue as to what "Make sure to secure your network with Tripwire Checkconfig or STIG options if its critical DMZ guets" means?

I guess I'll deal with that when I get to it...thanks anyway for the info.

[azn2kew]

Basically, if you have DMZ virtual machines, make sure go through standard security lockdown and best practices how to deploy ESX in DMZ environment. You can download the guide from www.vmware.com and click on "Resource" link it will include all the details you need. Tripwire CheckConfig is a free utility tool help lockdown most standard security for ESX 3.x environment which is very good. If you're very concern with more security lockdown, than using Department of Defense STIG UNIX RRS script to query and analyze more in depth. You can read more details on Security and Compliance box which Texiwill has extensive in depth with security.

Virtual Switches options:

vSwitch0 ->for Service Console and VMotion

vSwitch1 ->VMotion and Service Console

vSwitch2 ->Production Network

vSwitch3 ->DMZ Network

vSwitch4 ->Backup or iSCSI Network

If you found this information useful, please consider awarding points for "Correct" or "Helpful". Thanks!!!

Regards,

Stefan Nguyen

iGeek Systems Inc.

VMware, Citrix, Microsoft Consultant

[altonius]

I have something similar in our DMZ environmet, however we don't have an iSCSI SAN but a Fibre Attached SAN

pNIC0 (Onboard) and pNIC2 (PCI) attached to vSwitch0

pNIC1 (Onboard) and pNIC3 (PCI) attached to vSwtich1

pNIC4 (PCI) and pNIC5 (PCI) attached to vSwitch2

vSwitch0

Service Console

Set with pNIC0 as the primary and pNIC2 as the backup

VMotion

Set with pNIC2 as the primary and pNIC0 as the backup

vSwitch1

VM Trusted Network

pNIC1 and pNIC2 set as automatic / teamed

vSwitch2

VM DMZ Network

pNIC4 and pNIC5 set as automatic / teamed

It doesn't look like you utilise any VLANs. I would recommend putting your VMotion network on a different non-routed subnet, and if possible on a different VLAN.

Hope this helps (and doesn't add too much confusion)

[Texiwill]

Hello,

You have 5 basic networks on an ESX server:

Administrative Network --- Service Consoles and management tools like VC.... Should be firewalled from standard corporate network.

Console Network --- ILO/DRAC/Console KVMs - Access to which gives up everything....- Should be firewalled from standard corporate network and general administrative network + every other network

VMotion/SVMotion Network --- Every thing is sent in clear text. The one every hacker wants to access --- Should be its OWN private network preferably with its own pSwitches.

Storage Network --- iSCSI requires SC interaction but the data paths do not, and it is clear text protocol (iSCSI, NFS, FC) -- Should be 100% separate and firewalled network. SC must participate in this network when using iSCSI, but you can bridge the traffic through the firewall

VM Network --- Subsection:

DMZ Network --- Should be 100% separate and firewalled form all other networks. Note, due to other issue, it is recommended that all DMZ VMs be placed on their own LUNs attached to their own Hosts. Do not comingle DMZ and non-DMZ VMs on the same host (VMware whitepaper to this affect does exist).

Any other network -- should be separate from the Administrative, Console, VMotion and Storage Networks.

if VMs need to access storage then set up another network just for this preferably using their own SAN/NAS.... NPIV is the exception to this unfortunately.

Given the fact that VLANs (RFC 802.1q) does not guarantee security use of them depends on your levels of trust in VLANs. But basically you want 2 pNIC per Network for full redundancy, security, and performance. VM to VM traffic will be protected from VLAN attacks on the same vSwitch portgroup, but once outside the host that protection no longer exists.

6 pNICs is really not enough so you need to make some very tough security choices... Most people settle on using VLANs to help with this...

1 pNIC for SC, 1 pNIC for VMotion each on their own portgroup on the same vSwitch, requires VLANs

2 pNICs for Storage Network

2 pNICs for VM Network.... Note you should NOT have your DMZ systems on the same hosts as your production networks.

DMZ is the one network that should be 100% separate from your production and other networks. This includes the use of VLANs, but again. This depends on your level of trust.....

Best regards,

Edward L. Haletky

VMware Communities User Moderator

====

Author of the book 'VMWare ESX Server in the Enterprise: Planning and Securing Virtualization Servers', Copyright 2008 Pearson Education.

CIO Virtualization Blog: http://www.cio.com/blog/index/topic/168354

As well as the Virtualization Wiki at http://www.astroarch.com/wiki/index.php/Virtualization

[azn2kew]

Edward, that is very details replied you have and that should give her plenty of thoughts to design on her networking piece.

If you found this information useful, please consider awarding points for "Correct" or "Helpful". Thanks!!!

Regards,

Stefan Nguyen

iGeek Systems Inc.

VMware, Citrix, Microsoft Consultant

[AngelC2]

I believe I got enough examples/ideas to begin designing my environment the way I need to, so thanks to all that contributed.

One note, having 6 pNICs on each ESX host server is more than enough because every single host will have this setup so I don't see this being an issue since I won't initially have lots of VM's but will have plenty of ESX hardware and resources to go around :smileyblush:

Second Note, this one is for Stefan...don't EVER assume the name Angel implies female...ha ha...I get this a lot but I'm a male and Angel is a very common name in the Hispanic community for a male not a female. Now, when it comes to Angela, Angelina, etc. then its safe to assume its for a female. No hard feelings...its a common mistake :smileysilly:

[AngelC2]

No, I won't be initially using VLANs but I'm confident that I have enough pNIC's for redundancy and seperation/creation of vSwitches to come up with a network that will work for my environment using ideas presented in this thread.

I have a simple physical network environment, so I won't be needing to make my VMware network so complex with many vSwtches, etc. I'm keeping it simple and avoiding chaos :D:^0:0

Thanks for your input!

[azn2kew]

Good deal, sorry for mistaken cool name by the way Angel

If you found this information useful, please consider awarding points for "Correct" or "Helpful". Thanks!!!

Regards,

Stefan Nguyen

iGeek Systems Inc.

VMware, Citrix, Microsoft Consultant