Re: Cisco switch configuration problem? VM not con...

RichardWest · ‎02-14-2008

I have a Windows 2003 VM running as a DHCP server.

We I bring up other VM's, and the network is set to obtain an address from DHCP, they are unable to consistantly get an address from the server. Sometimes I have to issue the "ipconfig /release" and "ipconfig /renew" command 10 or 15 times before I get an address. Then if the VM reboots, I have to do this all over again.

A little background on my setup:

I have 4 servers running ESX 3.5
All servers have 6 pNICs - 2 are dedicated to VM network traffic using IP hash load balancing
pNICs are connected to a Cisco 3560 switch. The VM network traffic ports are setup in a trunk group - one trunk group for each ESX server (so 4 in my case)

One relationship that I have found is that when the client VM in one the same ESX server as the DHCP server VM, then everything works normally. However, if the DHCP server VM is on antoher ESX server then I have these problems. I know that VM's on seperate ESX servers can talk to each other over the network. I have tested this with several system that have static IP addresses.

*This makes me think that something on the Cisco switch is blocking the DHCP request packets from the client VM.</stro</p>

My swtichport configuration on the Cisco 3560 looks like this:

!
interface Port-channel2
description Port Channel to ESX01 - DATA
switchport trunk encapsulation dot1q
switchport mode trunk
!
interface GigabitEthernet0/1
switchport trunk encapsulation dot1q
switchport mode trunk
channel-group 2 mode on
!

Does anyone have any ideas on what could be causing my problems?

Chamon · ‎02-14-2008

We have Port fast on all of our switch ports that have the ESX hosts plugged into them. You may want to try this.

Have you checked for any speed and duplex mismatches? Do your ports on the Cisco blink orange?

If they are eventually getting IP addresses then I would say that the switch is not blocking the DHCP traffic.

Do you have one or multiple VLANS on this switch?

GBromage · ‎02-14-2008

Hi Richard!

Is your DHCP server a physcial machine, or is it running inside a VM? If it's on a VM, is there any correlation between machines on the same host vs. different hosts getting addresses?

I notice you've got 802.1q trunking enabled on your switches. Have you configured the VLAN ids on your port groups? Are VMs on the same VLAN as your DHCP server?

If they're on different subnets, are you using the switch as your default gateway, or do you have a separate router? Is that configured to allow DHCP forwarding?

-

I hope this information helps you. If it does, please consider awarding points with the 'Helpful' or 'Correct' buttons. If it doesn't help you, please ask for clarification!

RichardWest · ‎02-14-2008

The DHCP server is another VM running on an ESX server as well. As I mentioned in the original post there is a direct correlation between machines on the same host vs machines running on a different host. If the machines are on the same host then they get a DHCP address. If they are on another ESX host then they are unable to get an address. This is what makes me think it's a Cisco switch issue.

I do have 802.1q trunking enabled, and I have my Virtual Switches to use vlan 100. All of my ESX servers are setup to use vlan 100 for the VM data networks (the network that my VM's use for themselves to access network resources - Internet, DHCP, DNS, etc.). All of my VM's and ESX servers are on the same subnet, and same physical Cisco switch.

RichardWest · ‎02-14-2008

Chamon,

I had turned on Port fast, but I received a warning message on the router about possibly creating a routing loop, so I disabled it. That was going to be another question I was going to ask about, but it sounds like I can safely enable that.

All of the ports are solid green, except when they blink from traffic.

There are multiple vlans on the switch, however I have configured the virtual switches in ESX to tag the traffic with vlan id 100. Vlan 100 has been configured in the routers and the IP addresses that the systems are using are in the vlan IP range. All virtual switches are configured to use vlan 100 so the switch on separate ESX hosts should be able to talk to each other. In fact, when I setup my client to use a static IP address then VM's can talk to each other regardless of which ESX host they are running on.

GBromage · ‎02-14-2008

My guess then would be that it's not specifically DHCP that's the issue, but broadcasting.

Try this as a test: Have your DHCP server on one ESX server (I'll call this ESX1). On another host (ESX2), have a new machine prepared but not yet booted. Have two other machines running, one on ESX1 and one on ESX2 - booted up and running Network Monitor (or Wireshark or really any sort of packet analyzer). Boot the new machine, let it try to get an address and compare the output of the packet captures. Can both side see the DHCP request and response, or only one?

-

I hope this information helps you. If it does, please consider awarding points with the 'Helpful' or 'Correct' buttons. If it doesn't help you, please ask for clarification!

RichardWest · ‎02-15-2008

Chamon,

Do you enable portfast at the interface level or and the port-channel level?

When I issue the command "spanning-tree portfast" I receive the following error:

%Warning: portfast should only be enabled on ports connected to a single

host. Connecting hubs, concentrators, switches, bridges, etc... to this

interface when portfast is enabled, can cause temporary bridging loops.

Use with CAUTION

%Portfast has been configured on Port-channel2 but will only

have effect when the interface is in a non-trunking mode.

It seems from the last statement that this will not have an effect since the interface is configured as a trunk.

Chamon · ‎02-15-2008

No I think it is uplink fast or something along those lines. sorry for the

confusion.

RichardWest · ‎02-15-2008

GBromage,

I setup a test environment as you described. I had a server on host ESX1 running Wireshark, and a server on ESX2 running Wireshark. My DHCP server VM was running on ESX2.

After booting a client on ESX1 I see the "DHCP Discover" request on the ESX1 packet capture, but NOT on the ESX2 server.

When I move the "DHCP Client Test VM" to the ESX2 host I see the "DHCP Discover" request and the DHCP ACK from the DHCP server, and the client does get a DHCP assigned IP address. Futhermore, I do NOT see the DHCP Discover packet on the ESX1 host.

Therefore, in my previous statemetns where I said the DHCP process was inconsistant, I would like to revise that. After reflection I'm not sure that DHCP across the ESX hosts ever worked. I only thought it did becuase I was not paying attention to which ESX host a VM was running on.

It seems pretty clear from the Wireshark logs that the DHCP traffic is somehow being blocked from traveling from one ESX host to antoher.

Chamon · ‎02-15-2008

Does your switch have the dhcp relay option?

RichardWest · ‎02-15-2008

I'm not a Cisco expert by anymeans, but I'm unfamiliar with the option. I have tried setting an ip helper address on the vlan, but that did not seem to have any effect.

Is there a specfic Cisco command that you can share with me to try?

Chamon · ‎02-15-2008

Take a look at this link from cisco. I am not an expert either.

http://www.cisco.com/en/US/products/ps6350/products_configuration_guide_chapter09186a00804412bf.html

RichardWest · ‎02-15-2008

Thanks for the link Chamon. That discusses setting up the "ip helper-address", which I have tried with no luck.

FYI -- from what I understand, ip helper-addresses are only necessary if the DHCP server is located in a seperate VLAN than the client. They basically point to a server on another network telling the client where it can get an address from. The is necessary becuase the DHCP server would never receive any of the broadcast messages that the client is sending out becuase of the network segmentation.

RichardWest · ‎02-15-2008

Turns out its a known problem with the Intel VT Quad Port nic that we are using.

Hopefully an update will be released soon. I confirmed that this was an issue by removeing the vlan id from the vswitch, and reverting my Cisco switch configuration back to access ports and assigning the vlan id there. Once that was done DHCP clients were able to get addresses regardless which ESX server they were being hosted on.

dangsb · ‎02-15-2008

Your message

To: dan@gsb.com

Subject: New message: "Re: Cisco switch

configuration problem? VM not consistantly able to get IP address from DHCP

server"

Sent: Fri, 15 Feb 2008 14:14:18 -0600

did not reach the following recipient(s):