Guest VM's don't network

[tecvmware]

Hi,

Oddball question that I thought was just a one-off but I've now seen on a few host VI servers.

The problem is the server will host x number of VM's fine but then at some point if you add any more they refuse to network.

The environment is fairly simple - we have several standalone ESX 3.02 & 3.5 (generally unpatched) servers hosting Windows 2003 sp 2 hosts.

I've tried adding additional NIC's to guests so they DHCP (they just pick up a 169 internal address), tried booting Linux on CD ISO's to see if its OS related (same issue), and updating the VM tools. The NIC's with static addresses do see traffic (ie the nics are 'blue') they just don't connect to anything - as the ESX systems are standalone (ie not in Virtual Center environment) the host & guest(s) all share the same NIC.

Any ideas ? Are there any logs I can look at to see why the network isn't coming up ? Could it relate the the ESX systems changing IP address (they were moved to our data center) ?

Thanks in advance,

Raj.

[weinstein5]

What is the configuration of you virtual switchs? - Perhaps you can attach a screen capture of the networking configuration from ESX host - When you say they refuse to network are you referring to each other or to the outside world?

As long as the virtual switch is connected to a physical nic - the vm's that are connected through a virtual machine port group on that virtual switch they should see the network that physocal network can see -

[kjb007]

Take a look at your virtual switch properties, and make sure you have open ports. If you need to modify the number, you will need to reboot your server.

-KjB

[tecvmware]

Hi,

Attached is the network config - its very very simple - everything going out one NIC (Service Console + VM's). This particular box is ESX 3.5 but it also happens on a 3.02 system.

The default vswitch port config is for 56 ports - the maximum number of VM's on these standalone servers is about 8.

The only thing I can think of is running the esxcfg-vswif -d vswif0 to remove and esxcfg-vswif -a vswif to add it back in again in order to handle the IP address change when it shifted from my office to the datacenter has somehow munged something internally.

Cheers,

Raj.

[weinstein5]

Is there a DHCP server on the network that vmnic0 is on? If you hard code on IP address on ofe your NIC say 10.1.45.X - can you ping the Service Console IP? Also you can change the vswif IP from the VI Client -

[tecvmware]

Hi,

Yeah we have a dhcp server - if I set a new VM guest NIC to dhcp it doesn't pick up a valid address - just an APIP (ie a 169.x.x.x. address).

The other running VM's are fine - its just any new VM I try to add to these servers.

It has to be something simple but for the life of me I can't find anything to debug the issue or indicate why newly created VM's won't talk to the network.

Raj.

[kjb007]

Are you able to access the service console IP address? Are the VM's on the same network segment as the service console?

-KjB

[tecvmware]

Hi,

Yes - the service console is accessible via the VI tool & via SSH. The VM's are on the same network segment. All pretty straightforward - thats why its such a frustrating issue.

-Raj.

[kjb007]

Can you post your switch config from your service console, esxcfg-vswitch -l, as well as your esxcfg-vswif -l

One more thing to check, do you have beacon probing turned on in your portgroup settings?

-KjB

Message was edited by: kjb007 : Added beacon probing

[tecvmware]

Hi,

No beaconing enabled.

Configs below -

root@server root# esxcfg-vswitch -l

Switch Name Num Ports Used Ports Configured Ports MTU Uplinks

vSwitch0 64 7 64 1500 vmnic0

PortGroup Name VLAN ID Used Ports Uplinks

VM Network 0 3 vmnic0

Service Console 0 1 vmnic0

root@server root# esxcfg-vswif -l

Name Port Group IP Address Netmask Broadcast Enabled DHCP

vswif0 Service Console 10.1.45.49 255.255.255.0 10.1.45.255 true false

[kjb007]

Can you re-install the vmware tools?

-KjB

[tecvmware]

Just ran up a new system and re-installed the tools - no joy. I get the same APIP address or if I set a static address instead of DHCP it just won't talk to anything.

Any logs I can check ?

Cheers,

Raj.

[kjb007]

Do you have port security on the switch? If you have multiple VM's on that vm network with static addresses, can they talk to the other vm's? Meaning, can the ESX host vm's communicate with each other, if not the rest of your network?

-KjB

[tecvmware]

Yes - the VM's that work on that host can ping each other and the rest of the network aok.

Anything new just won't see the network.

Cheers

Raj.

[kjb007]

Have you checked your vswitch properties to see how many ports are on the switch? If your existing vm's can ping each other and the rest of the network, then you may have run out of uplinks.

-KjB

[kjb007 · Accepted solution]

Sorry, I already asked that. If you have a running vm, and take it down, and start a new vm. Does the new nic work?

Have you checked port security on the switch?

-KjB

[tecvmware]

I tried a few more tests -

Running up a new Service Console & VM Network and attaching the new VM to it - no change.

Shutting down a working VM and running up the test VM - no change.

We do run MAC based switchport security - I'll get it disabled and try it again.

[weinstein5]

I will bet MAC based switch based security is it - remember every time you create a new VM a new unique MAC address will be created for that new VM - so unless you update the allowed MAC addresses on the port security no traffic will come through -

Also I can not remember uyou are connecting to a virtual switch with more than one physical NIC - but if you are you will have to configure the ports the physical NICs connect to depending on your load balancing method and for NIC failover -

[tecvmware]

Looks like KjB was spot on - we had 10 MAC addresses against the physical NIC on the switch - once we extended this the VM networked AOK.

Something so simple and yet so frustrating!

Cheers,

Raj.

[tecvmware]

Actually one more thing - is there any point having MAC based switchport security ? Given you may VMotion at any time to and from any ESX box doesn't that mean MAC security becomes more of a hinderence ?

Cheers,

Raj.