Need advice on fully redundant network setup

[pauska]

Hi again!

We have 2xHP DL380's with 6 pNic's installed, and we aim to get fully redundant SAN+network setup. The SAN is already designed for redunancy, but I'm very unsure about the network bit..

I have 2xHP ProCurve 2810-24G switches, each has a 2gbit trunk(lacp) between them and 2gbit trunk(lacp) out to the core switch. I've read the 2810 manual wich mentions stacking - but this seems to be a feature for ease of management - not redundancy? reference:

I've seen in earlier threads (cant find them) about people choosing the more expensive Cisco 3750 switches because of clustering - is it impossible for me to achieve the same results with my switches? Do I have to just rely on a failover setup instead of loadbalancing across switches?

[nappyrat]

We have 2 cisco 3750's in a stack. the benefit of it is that when you have them setup in a stack, you can create port channels across the switches. That is, you can plug a machine or storage that has 2 nics into port 1 on switch 1 and port 1 on switch 2. you create a lacp port channel out of those 2 nics. then, not only do you have load balancing across both the swtiches, but redundancy if one switch fails.

the only down side is when doing any type of configuration or upgrade that requires resetting one of the switches, you usually have to reset both because if the configs/versions aren't exact they won't join the stack.

I don't know much about those hp switches but i'm willing to bet you can't create a port channel across devices the way you have them setup.

[azn2kew]

Your hardware should provide decent network redundant for you as well as 6 pNICs involved for ESX hosts. Each port group should have dual pNICs connected to your dual HP ProCurve switch so that helps a lot for your concerns. I would configure NICs like this:

1. pNIC1->SC/VMotion standby

2. pNIC2->VMotion/SC standby

3. pNIC3-4->VM Network

4. pNIC5-6->DMZ Network (if required)

Or you can have all 3 ports redundant with dual NICs

1. pNIC1-2->SC

2. pNIC3-4->VMotion

3. pNIC5-6->VM Network

Read the attach guides on combination of network design metrix. There are different combinations and depends which one you like the most and I would prefer to have spare NICs if possible.

If you found this information useful, please consider awarding points for "Correct" or "Helpful". Thanks!!!

Regards,

Stefan Nguyen

iGeek Systems Inc.

VMware, Citrix, Microsoft Consultant

[pauska]

Thanks for the informative posts.

Stefan, im going to go for your first pnic setup as I do have a DMZ and possible a public zone in the future. This should be easy to set up with VLAN and LACP.

My question is though - how should I wire this to the switches? Do I have to keep the load-balancing nics on a single physical switch, and only get full redunancy on vmotion/sc with standby nic?

[azn2kew]

If you have the luxury to do load balancing on both physical and virtual switches, than it would be great and it gives you the comfort level of neither failured you still have something in placed. You can go to www.vmware-land.com for tons of designs and practices for networking and extracted some of them here.

Networking - General:

VMware Virtual Networking Concepts -

VI3 Networking - Concepts and Components (VMworld 2007) -

VI3 Networking - Advanced Configurations and Troubleshooting (VMWorld 2007) - New - 5/1/08

High Performance Virtualized I/O in 10 Gigabit Ethernet Era (VMWorld 2007) - New - 5/1/08

Networking Virtual Machines -

Networking Scenarios & Troubleshooting -

ESX3 Networking Internals -

High Performance ESX Networking -

Network Throughput in a Virtual Infrastructure -

How To Configure Networking from the Service Console Command Line -

Integrating Virtual Machines into the Cisco Data Center Architecture -

Severe Network Performance Issues -

ESX Server 3.0.2 Ping Latency -

How to Configure Windows Server 2003 as a Router -

HA excessive number of open TCP/UDP ports required -

Troubleshooting Slow Virtual Web Servers - New - 5/1/08

Performance Comparison of Virtual Network Devices - New - 5/1/08

Networking Performance in VMware ESX Server 3.5 - New - 5/1/08

Server Virtualization - Network Implications and Best Practices - New - 5/1/08

Networking - Firewall:

Firewall Configurations for Backup Clients on ESX Server 3 - New - 5/1/08

3rd party software and the new Service Console Firewall -

Firewall and Netbackup -

Esxcfg-firewall -

Networking - vSwitch:

Rename a vSwitch - and

Enabling Netflow on Virtual Switches - New - 5/1/08

Configuring and Troubleshooting N-Port ID Virtualization - New - 5/1/08

Physical Switch Load Balancing -

Networking - NIC:

Network cards - Used Auto or Fixed Speed and Duplex - http://www.vmware.com/community/thread.jspa?messageID=568366

How to identify a physical NIC - http://www.vmware.com/community/thread.jspa?messageID=662139

Which physical NIC is being used by a VM - http://www.vmware.com/community/thread.jspa?messageID=591252

Setting gigabit NIC's to Auto/Auto - http://www.vmware.com/community/thread.jspa?messageID=568366

Vmnic renumbering - http://www.vmware.com/community/thread.jspa?messageID=477265 and http://www.vmware.com/community/thread.jspa?messageID=551433

Changing Service Console NIC IP Address - http://www.vmware.com/community/thread.jspa?threadID=81427 and http://kb.vmware.com/kb/4309499

Change service console NIC's - http://www.vmware.com/community/thread.jspa?messageID=565489

Second Service Console NIC - http://www.vmware.com/community/thread.jspa?messageID=536518

Which NIC is used in cold migration - http://www.vmware.com/community/thread.jspa?messageID=555404

Duplex Mismatch Primer - http://www.happyrouter.com/content/view/32/1/

Multi-NIC Performance in ESX 3.0.1 and XenEnterprise 3.2.0 - http://www.vmware.com/pdf/Multi-NIC_Performance.pdf

Service Console and Vmotion NIC sharing - http://www.vmware.com/community/thread.jspa?messageID=146348

Why do network hint ranges for NIC's using VLAN tagging keep changing - http://www.vmware.com/community/thread.jspa?messageID=678394 and http://www.vmware.com/community/thread.jspa?messageID=503421

VM Summary Page General showing other NIC IP Address - http://www.vmware.com/community/thread.jspa?threadID=84784

Configure multiple IP addresses to a NIC in Linux - http://handsonhowto.com/virt.html

Configure a static MAC address for a VM - http://virtrix.blogspot.com/2007/04/vmware-configuring-static-mac-address.html

Virtual MAC Tool (configuration) - http://www.run-virtual.com/?page_id=173

NIC Teaming 802.3ad - http://www.vmware.com/pdf/esx2_NIC_Teaming.pdf

ESX 3.0.1/NIC Teaming/802.3ad/Aggregrating Bandwidth - http://www.vmware.com/community/thread.jspa?messageID=506538

ESX Server Requirements for Link Aggregation - http://kb.vmware.com/selfservicekb/1001938

802.3ad with IP Hash in ESX will increase bandwidth - http://www.vmware.com/community/thread.jspa?messageID=682252

NIC Teaming Load Balancing Policy - http://www.vmware.com/community/thread.jspa?messageID=463074

Can a VM use more then one physical NIC on a vSwitch - http://www.vmware.com/community/thread.jspa?messageID=632932

Bonding NIC's on a VM - http://www.vmware.com/community/thread.jspa?messageID=561718

Aggregate multiple NIC's into a single pipe - http://www.vmware.com/community/thread.jspa?threadID=83099

Changing the MAC address of a virtual machine - http://kb.vmware.com/kb/507

VI Client loses connectivity to the ESX Server Host after you add a new network adapter - http://kb.vmware.com/kb/2243

Networking - VLAN/802.1Q:

VMware ESX Server 3 802.1Q VLAN Solutions - http://www.vmware.com/pdf/esx3_vlan_wp.pdf

Configuring VLAN's in VMware Infrastructure 3 - http://searchvmware.techtarget.com/tip/0,289483,sid179_gci1280449,00.html

VLAN Configuration on VI3: VST, EST VGT Tagging Tips - http://searchvmware.techtarget.com/tip/0,289483,sid179_gci1283036,00.html New - 5/1/08

VLAN Trunking - Do Rewards Outweigh the Risks - http://www.vmware.com/community/thread.jspa?threadID=29466

ESX Server, NIC Teaming and VLAN Trunking - http://blog.scottlowe.org/2006/12/04/esx-server-nic-teaming-and-vlan-trunking/

If you found this information useful, please consider awarding points for "Correct" or "Helpful". Thanks!!!

Regards,

Stefan Nguyen

iGeek Systems Inc.

VMware, Citrix, Microsoft Consultant

[InfoStewards]

I would stay away from stacking as a means of redundancy. Use it for port expansion and ease-of-management instead.

If you're using the switches for layer2 only, be sure to plumb your VLANs across that LACP xconnect and also out to the LACP to the cores. That will allow you to lose either switch and still keep both ESX servers online.