physical NIC mappings wrong after upgrade from 3.5 Update 1 to 3.5 Update 2

[radman]

I have a dual-homed host, attached to two separate subnets (two vSwitches). It was working fine with ESX 3.5 Update 1. Now that I've installed Update 2, however, my VMs still intercommunicate on both virtual switches but they can't reach the outside world on one of the subnets.

I tried logging into the console, and ifconfig -a only shows one interface configured with an IP address (on the subnet which is working fine), called vswif0. vmnic0-3 all show "up" but no IP address. Other than "lo" those are the only interfaces reported.

If I go into VirtualCenter, and look at Configuration/Networking, it shows two virtual switches, connected to vmnic0 and vmnic2, and the connectivity looks right. If I go into Properties/Network Adapters for each, they both show "Connected", with reasonable speeds (one is Gb, one is 100Mb, which is as it should be). The nVidia NForce adapter shows "forcedeth" as the driver, the one that isn't working (82571EB) shows "e1000".

If I look at Configuration/Network Adapters, it shows reasonable-looking information (although the broken vSwitch1 doesn't show any "Observed IP ranges").

I notice that on the console, in the /etc/sysconfig/network-scripts directory, there are the following ifcfg-* scripts:

ifcfg-eth1

ifcfg-eth2

ifcfg-eth3

ifcfg-vswif0

This makes sense - I actually have two unused network adapters. Also, the comments correctly reflect the right number of types of controllers (2 x nVidia, 2 x 82571EB), but all except for ifcfg-vswif0 have "ONBOOT=no", which seems suspicious. One of the 82571EB should have ONBOOT=yes, presumably, and I'm not sure about the naming. Should it be ifcfg-vswif1 or something? What went wrong, and how should I fix it? How do I determine the MAC address of the unplumbed interface?

There are a bunch of errors in /var/log/messages from boot saying things like:

ifup: ./ifup: line 7: Console: command not found

ifup: /etc/sysconfig/network-scripts/ifup-post: line 7: Console: command not found

ifup: /etc/sysconfig/network/scripts/ifup-aliases: line 7: Console: command not found

Any help much appreciated. I don't know where to go from here.

[weinstein5]

Remember the service console is a virtual machine running on your esx host so we need to see what the vmkernel see - from the service console can you provide the ourpur of the following commands:

esxcfg-nics -l this will list the nics the vmkernel sees

esxcfg vswitch -l this will list the configured virtual switches

esxcdg-vswif -l - lists the ocnfigured interfaces for the service console

If you want to see some more of these esxcsfg command checkou -

[radman]

Remember the service console is a virtual machine running on your esx host<div>Yes, but it's also the environment used to bootstrap ESX in the first place, right? So what's in there is what creates the configuration that ESX inherits.

</div>

In any case, the commands you suggest provide output that aligns with the data I presented (much more concisely, thanks :

# esxcfg-nics -l
Name PCI Driver Link Speed Duplex MTU Description
vmnic3 80:0a.00 forcedeth Down 0Mbps Half 1500 nVidia Corporation nVidia NForce Network Controller
vmnic1 02:00.00 e1000 Down 0Mbps Half 1500 Intel Corporation 82571EB Gigabit Ethernet Controller
vmnic0 00:0a.00 forcedeth Up 1000Mbps Full 1500 nVidia Corporation nVidia NForce Network Controller
vmnic2 02:00.01 e1000 Up 100Mbps Full 1500 Intel Corporation 82571EB Gigabit Ethernet Controller

# esxcfg-vswitch -l

Switch Name Num Ports Used Ports Configured Ports MTU Uplinks
vSwitch0 32 9 32 1500 vmnic0

PortGroup Name VLAN ID Used Ports Uplinks
VM Network 0 5 vmnic0
Service Console 0 1 vmnic0
VMkernel 0 1 vmnic0

Switch Name Num Ports Used Ports Configured Ports MTU Uplinks
vSwitch1 64 5 64 1500 vmnic2

PortGroup Name VLAN ID Used Ports Uplinks
Interconnect Network0 3 vmnic2

# esxcfg-vswif -l
Name Port Group IP Address Netmask Broadcast Enabled DHCP
vswif0 Service Console 172.16.0.26 255.255.0.0 172.16.255.255 true false

So I understand why vmnic1 and vmnic3 are down - they're not physically connected to anything. But why isn't there a vswif that corresponds to vmnic2? There was before the upgrade...

I've done a little more experimenting and the problem isn't as bad as I first thought. vSwitch1 is working fine to the outside world. It just can't contact the Service Console, so I think the only issue is that the Service Console needs an interface that maps to vmnic2.

If you want to see some more of these esxcsfg command checkou -

Unfortunately your message was truncated. I've used some but never seen a comprehensive guide.

Thanks,

Bob

P.S. It's really too bad there's no "preformat" formatting option for pasting command output I tried manually inserting &lt;pre&gt; &lt;/pre&gt; in the Plain Text window but even that fails.

</div>

[Schorschi]

This is not specific to 3.5 Update 1 to Update 2, can happen on any 3.5 version. To avoid this, just use --device option for network line in kickstart, or post build run the reorder.py script that is on the yellow-brick blog. Based on how the NIC drivers load, the lowest order NIC on the PCI bus is not alway the one bound to the service console vswif interface, which should always be vmnic0, but is not. The build makes the assumption that vmnic0 is always the lowest order NIC, so unless the --device option is used, Anaconda/Kickstart and VMware ESX vmkernel do not synchronize driver load, so you get this hang up. We now run reorder script on every box, and reboot the box, this clears the issue up for us.

[radman]

post build run the reorder.py script that is on the yellow-brick blog.

Thanks, but can you provide a specific URL? Searching for reorder.py and yellow-brick turns up nothing and yellow-brick is far too generic a search term

My problem isn't the ordering of the interfaces, it's simply that I want my Service Console to be multi-homed with both interfaces plumbed. I can't even seem to plumb any other interfaces manually - not sure what interface names to use. If I knew the proper interface names I could fix the network-scripts myself.

[mike_laspina]

Hello,

Just an observation. You indicated that all nics were in a link up state. But esxcfg-nics reports down for 1 and 3?

[mike_laspina]

It would not be prudent to manually change the networks-scripts

If you wish to change the physical to name map you can backup the /etc/vmware/esx.conf file and change the names there.

e.g.

/net/pnic/child[0000]/mac = "00:02:b3:a2:a8:ae"

/net/pnic/child[0000]/name = "vmnic0"

/net/pnic/child[0000]/virtualMac = "00:50:56:52:a8:ae"

/net/pnic/child[0001]/mac = "00:11:25:8f:94:2c"

/net/pnic/child[0001]/name = "vmnic1"

/net/pnic/child[0001]/virtualMac = "00:50:56:5f:94:2c"

/net/pnic/child[0002]/mac = "00:11:25:8f:94:2d"

/net/pnic/child[0002]/name = "vmnic2"

/net/pnic/child[0002]/virtualMac = "00:50:56:5f:94:2d"

[radman]

I didn't mean to imply that all nics are up. I meant that all the nics that are connected are in fact up. It's correct that 1 and 3 should be down - they are disconnected.

My problem isn't ordering or naming of nics. My problem is that not all nics are plumbed for the Service Console - only nic0 is. I want nic2 plumbed as well for the Service Console. Any clues on how to do that? Or why an upgrade from Update 1 to Update 2 would cause nic2 to become unplumbed?

Thanks,

Bob

[mike_laspina · Accepted solution]

esxcfg-vswitch --add-pg='Service Console 2' vSwitch1

adds a port group for a console if to connect to.

esxcfg-vswif -a -i 172.16.0.1 -n 255.255.0.0 -p 'Service Console 2' vswif1

adds the if to the switch

[weinstein5]

For more information on the esxcfg commands look here -