Re: ESX Hosts disconnected from VC, "unable to con...

NotAvailable · ‎11-17-2008

Hi all,

After installing the ESX-Updates we are having the Problem that 2 of 3 ESX3Upd2 Hosts are disconnected from VC and can not be connected any more. The VC error is:

Unable to connect to the specified host (esx1.domain.intern). This might be because the host is not available on the network, there's a network configuration problem, or the management services on the host are not responding.Network seems to be OK, as ping, VI Client direct connect and ssh are OK.

ESX1 and ESX2 are disconnected, ESX3 is fine. VC is running on a VM.

We tried the following:

- Restarting mgmt-vmware and vmware-authentication

- Rebooting the hosts

- Uninstalling vpxa and aam from one of the hosts (-> vpxa is reinstalled automatically when I try to reconnect from VC)

- Removing one host from VC and connect again

When I connect host from VC, the task first fails with the error mentione, then it gives the connect dialog where I enter hostname, username (root ) and password (credentials are fine, no error there), then I verify "Host Summary and VM Location, and after completing it needs a couple of seconds just to return the error again.

/var/log/messages:

Nov 17 10:29:44 esx2 vmware-hostd[17646]: Accepted password for user vpxuser from 127.0.0.1

Nov 17 10:29:44 esx2 last message repeated 3 times

Nov 17 10:29:46 esx2 watchdog-hostd: '/usr/sbin/vmware-hostd -u' exited after 2253 seconds

Nov 17 10:29:46 esx2 watchdog-hostd: Executing cleanup command '/usr/sbin/vmware-hostd-support'

Nov 17 10:29:46 esx2 watchdog-vpxa: '/opt/vmware/vpxa/sbin/vpxa' exited after 2253 seconds

Nov 17 10:29:46 esx2 watchdog-vpxa: Executing '/opt/vmware/vpxa/sbin/vpxa'

Nov 17 10:29:46 esx2 watchdog-hostd: Executing '/usr/sbin/vmware-hostd -u'

Nov 17 10:29:53 esx2 modprobe: modprobe: Can't locate module char-major-14

Nov 17 10:29:53 esx2 modprobe: modprobe: Can't locate module block-major-2

Nov 17 10:29:53 esx2 last message repeated 6 times

Nov 17 10:29:53 esx2 modprobe: modprobe: Can't locate module char-major-14

Nov 17 10:29:53 esx2 modprobe: modprobe: Can't locate module block-major-2

Nov 17 10:29:53 esx2 last message repeated 6 times

Nov 17 10:29:54 esx2 vmware-authd(pam_unix)[17949]: authentication failure; logname= uid=0 euid=0 tty= ruser= rhost= user=root

Nov 17 10:29:57 esx2 modprobe: modprobe: Can't locate module block-major-2

Nov 17 10:29:57 esx2 last message repeated 6 times

Nov 17 10:29:57 esx2 modprobe: modprobe: Can't locate module char-major-14

Nov 17 10:30:00 esx2 vmware-hostd[17949]: Accepted password for user root from 127.0.0.1

Nov 17 10:31:08 esx2 vmware-hostd[17949]: Accepted password for user root from 127.0.0.1

Nov 17 10:31:09 esx2 passwd(pam_unix)[18074]: password changed for vpxuser

Nov 17 10:31:09 esx2 vmware-authd(pam_unix)[17949]: authentication failure; logname= uid=0 euid=0 tty= ruser= rhost= user=root

Nov 17 10:31:11 esx2 vmware-hostd[17949]: Accepted password for user vpxuser from 127.0.0.1

Nov 17 10:31:11 esx2 last message repeated 2 times

Nov 17 10:31:13 esx2 watchdog-hostd: '/usr/sbin/vmware-hostd -u' exited after 87 seconds

Nov 17 10:31:13 esx2 watchdog-hostd: Executing cleanup command '/usr/sbin/vmware-hostd-support'

Nov 17 10:31:13 esx2 watchdog-vpxa: '/opt/vmware/vpxa/sbin/vpxa' exited after 87 seconds

Nov 17 10:31:13 esx2 watchdog-vpxa: Executing '/opt/vmware/vpxa/sbin/vpxa'

Nov 17 10:31:13 esx2 watchdog-hostd: Executing '/usr/sbin/vmware-hostd -u'

Nov 17 10:31:20 esx2 modprobe: modprobe: Can't locate module char-major-14

Nov 17 10:31:20 esx2 modprobe: modprobe: Can't locate module block-major-2

Nov 17 10:31:20 esx2 last message repeated 6 times

Nov 17 10:31:20 esx2 modprobe: modprobe: Can't locate module char-major-14

Nov 17 10:31:20 esx2 modprobe: modprobe: Can't locate module block-major-2

Nov 17 10:31:20 esx2 last message repeated 6 times

Nov 17 10:31:21 esx2 vmware-authd(pam_unix)[18195]: authentication failure; logname= uid=0 euid=0 tty= ruser= rhost= user=root

Nov 17 10:31:24 esx2 modprobe: modprobe: Can't locate module block-major-2

Nov 17 10:31:24 esx2 last message repeated 6 times

Nov 17 10:31:24 esx2 modprobe: modprobe: Can't locate module char-major-14

/var/log/vmkernel:

Nov 17 10:29:53 esx2 vmkernel: 2:13:31:18.485 cpu2:1041)Config: 416: "Enabled" = 1, Old Value: 1, (Status: 0x0)

Nov 17 10:29:53 esx2 vmkernel: 2:13:31:18.658 cpu0:1024)Host: 2655: irq 4 vector 0x31 (host 0x41)

Nov 17 10:29:53 esx2 vmkernel: 2:13:31:18.658 cpu0:1024)Host: 2587: irq 4 vector 0x31 (host 0x41)

Nov 17 10:29:53 esx2 vmkernel: 2:13:31:18.658 cpu0:1024)Host: 2599: irq 4 happened while being masked

Nov 17 10:29:53 esx2 vmkernel: 2:13:31:18.802 cpu0:1024)Host: 2655: irq 4 vector 0x31 (host 0x41)

Nov 17 10:29:53 esx2 vmkernel: 2:13:31:18.802 cpu0:1024)Host: 2587: irq 4 vector 0x31 (host 0x41)

Nov 17 10:29:53 esx2 vmkernel: 2:13:31:18.802 cpu0:1024)Host: 2599: irq 4 happened while being masked

Nov 17 10:29:57 esx2 vmkernel: 2:13:31:22.556 cpu0:1024)Host: 2655: irq 4 vector 0x31 (host 0x41)

Nov 17 10:29:57 esx2 vmkernel: 2:13:31:22.556 cpu0:1024)Host: 2587: irq 4 vector 0x31 (host 0x41)

Nov 17 10:29:57 esx2 vmkernel: 2:13:31:22.556 cpu0:1024)Host: 2599: irq 4 happened while being masked

Nov 17 10:29:57 esx2 vmkernel: 2:13:31:22.572 cpu3:1039)Config: 416: "VMOverheadGrowthLimit" = -1, Old Value: -1, (Status: 0x0)

Nov 17 10:31:20 esx2 vmkernel: 2:13:32:45.723 cpu2:1041)Config: 416: "Enabled" = 1, Old Value: 1, (Status: 0x0)

Nov 17 10:31:20 esx2 vmkernel: 2:13:32:45.901 cpu0:1024)Host: 2655: irq 4 vector 0x31 (host 0x41)

Nov 17 10:31:20 esx2 vmkernel: 2:13:32:45.901 cpu0:1024)Host: 2587: irq 4 vector 0x31 (host 0x41)

Nov 17 10:31:20 esx2 vmkernel: 2:13:32:45.901 cpu0:1024)Host: 2599: irq 4 happened while being masked

Nov 17 10:31:20 esx2 vmkernel: 2:13:32:46.049 cpu0:1024)Host: 2655: irq 4 vector 0x31 (host 0x41)

Nov 17 10:31:20 esx2 vmkernel: 2:13:32:46.049 cpu0:1024)Host: 2587: irq 4 vector 0x31 (host 0x41)

Nov 17 10:31:20 esx2 vmkernel: 2:13:32:46.049 cpu0:1024)Host: 2599: irq 4 happened while being masked

Nov 17 10:31:24 esx2 vmkernel: 2:13:32:49.716 cpu0:1024)Host: 2655: irq 4 vector 0x31 (host 0x41)

Nov 17 10:31:24 esx2 vmkernel: 2:13:32:49.716 cpu0:1024)Host: 2587: irq 4 vector 0x31 (host 0x41)

Nov 17 10:31:24 esx2 vmkernel: 2:13:32:49.716 cpu0:1024)Host: 2599: irq 4 happened while being masked

Nov 17 10:31:24 esx2 vmkernel: 2:13:32:49.783 cpu2:1041)Config: 416: "VMOverheadGrowthLimit" = -1, Old Value: -1, (Status: 0x0)

Any help would greatly be appreciated...

Thanks in advance!

Marcus

pdarcysgi · ‎11-17-2008

Hi

Have you upgraded the vc to update 2 or 3 yet? This happens when the VC isn't updated first?

p

Karunakar · ‎11-17-2008

Can you also try to perform the below steps.

I hope they could be some elp.

In ESX go to /tmp, and check if a folder named "vmware-root" is available, if not, then create the folder, its an empty folder.

Then try to connect the machine to VC.

If the above does not work, try to follow the below steps.

> Copy the vpx-upgrade-esx-7 or 6****** from the upgrade folder in the VC folder to /tmp on the ESX server

(By default the script is located in C:\program files\vmware\vmware virtualcenter\upgrade )

> run the command " sh vpx-upgrade-esx-7 or 6****** " in ESX

Then restart mgmt-vmware process as "/etc/init.d/mgmt-vmware start"

And then after the hostd process start, wait for one minute, and try to add the machine to vc.

Let me know if any results.

-Karunakar

NotAvailable · ‎11-17-2008

Hi p

Both ESX and VC have been updated to Update 2 some weeks ago (and went fine)

On ESX, the command cat /proc/vmware/version shows

VMware ESX Server 3.5.0 , built on Aug 26 2008

vmkernel build: 113339, vmkcall: 45.0 driver interface: 8.0 kernel: 61.0

vmnixmod build: 113339, interface: 2850028248

vmksysinfo version: 83521b8d3a8bccfb7444d2be30127713

Loadable module version info:

vmkernelID: 0x1fa209c0

vmklinux build 113339: Built on: Aug 26 2008

ioat build 82663: 0.32

ata_piix build 82663: 1.05

bnx2 build 113339: 1.5.10b.1

aacraid_esx30 build 82663: 1.1-5[d-8930]custom-IBM

tg3 build 110268: tg3.c:v3.81c (September 18, 2007)

e1000 build 95350: 7.3.15

lvmdriver build 113339: 3.16, built on: Aug 26 2008

vmfs3 build 113339: 4.31, built on: Aug 26 2008

etherswitch build Built on: Aug 26 2008

shaper build Built on: Aug 26 2008

tcpip build 113339: Built on: Aug 26 2008

cosShadow build 113339: Built on: Aug 26 2008

migration build 113339: protocol: 5.0 vmk interface: 3.0

nfsclient build 113339: Built on: Aug 26 2008

deltadisk build 113339: Built on: Aug 26 2008

vmfs2 build 113339: 2.52, built on: Aug 26 2008

iscsi_mod build

The ESX Host in VC is displayed with Version 120512.

Help -> About VMware Infrastructure (in VC) shows VirtualCenter Build 104215

VC has not been updated with the latest host updates. (But actually I wonder why esx3 is still ok then...) - and the list of updates in ESX does not show ESX Update 3.

NotAvailable · ‎11-20-2008

Hi all

Just in case someone else gets these errors too, here the solution:

Cause:

The customer tried to import a Vmware Server VM into Vi3. But instead of using Converter, he just coied the files on VMFS SAN Volumes and clicked "Import xyz.vmx" in Storage Browser. Looks like the VM was tried to be imported but failed (cause it was in VMware SErver format). This made the hostd fail...

Solution:

After reboot ESX, we could connect with VI Client directly to the esx host and destroy the VM. Then reconnect to VC and everything is fine.

Thanks for your help anyway!!

Marcus

All

ESX Hosts disconnected from VC, "unable to connect to the specified host"