esx server suddenly not responding in virtual center and vi client

[raoulst]

Hi,

all of a sudden one of my hosts in a vi3 cluster stopped responing in virtual center. I tried to disconnect and reconnect, but that did not work.

When trying to reconnect to VC, I always get the error:

"Unable to access the specified host. It either does not exist, the server software is not responding, or there is a network problem."

When I try to directly connect to the Server via the VI Client I get the message:

"Connection failed".

The network ports seem to work.

In /var/log/messages, I found out, that the snmpd complains about the following:

: VmomiClient::Init: :514 Error connecting to hostd-vmdb service instance.

: Could not init vmomi client to vmware-hostd.

: GetNumCPu: Could not acquire vmomi client to vmware-hostd.

The VMs seem to be still running. I applied all patches and restarted

mgmt-vmware

and

vmware-vmkauthd

any ideas what might be wrong?

raoulst

Message was edited by:

raoulst

Message was edited by:

raoulst

[VirtualNoitall]

Hello,

Have a look at the following log files on your ESX host:

/var/log/vmware/hostd.log

/var/log/vpx/vmware/vpxa.log

Look for errors, failed, or anything the sounds out of place.

By chance did you run out of disk space on your ESX host?

[raoulst]

thank you,

also cmware-cmd-l and vdf do timeout.

it does not seem, like i did run out of disk space.

the last hostd.log entries are:

\[2007-07-07 00:58:48.067 'DatastoreBrowser' 3076456576 verbose] ha-legacy-envmgr-datastorebrowser::IntializeCOSDirectory: COS path: /vmimages/

\[2007-07-07 00:58:48.067 'DatastoreBrowser' 3076456576 verbose] ha-legacy-envmgr-datastorebrowser::Constructor

\[2007-07-07 00:58:48.067 'Solo' 3076456576 info] Micro web server port: 9080

\[2007-07-07 00:58:48.067 'App' 3076456576 panic] Application error: Address already in use

\[2007-07-07 00:58:48.067 'App' 3076456576 panic] Backtrace generated:

\[00] eip 0x11cb34e

\[01] eip 0x1121f79

\[02] eip 0x10dc215

\[03] eip 0x11e408d

\[04] eip 0x11e26a6

\[05] eip 0x11e1f70

\[06] eip 0x836be89

\[07] eip 0x82f93ed

\[08] eip 0x8305d24

\[09] eip 0x71f79a

\[10] eip 0x8090ed1

the vpxa.log does also seem to have problems

the last entries in

/var/log/vmware/vpx/vpxa.log

are...

\[2007-07-07 00:03:39.746 'App' 52841392 info] VpxaHalNfcServiceHostagent::NfcGetVmFiles, vmId = 2

\[2007-07-07 00:03:39.945 'App' 52841392 info] \[VpxLRO] -- FINISH task-internal-2383 -- -- \[vpxa:nfcGetVmFiles]

\[2007-07-07 00:03:52.261 'App' 52841392 info] \[VpxLRO] -- BEGIN task-internal-2384 -- -- \[vpxa:nfcGetVmFiles]

\[2007-07-07 00:03:52.261 'App' 52841392 info] VpxaHalNfcServiceHostagent::NfcGetVmFiles, vmId = 21

\[2007-07-07 00:03:52.358 'App' 52841392 info] \[VpxLRO] -- FINISH task-internal-2384 -- -- \[vpxa:nfcGetVmFiles]

\[2007-07-07 00:05:28.401 'App' 52841392 info] \[VpxLRO] -- BEGIN task-internal-2385 -- -- \[vpxa:setServer]

\[2007-07-07 00:05:28.401 'App' 52841392 info] \[VpxaInvtHost] Server IP has been cleared by 131.130.229.132

\[2007-07-07 00:05:28.427 'App' 52841392 info] \[VpxLRO] -- FINISH task-internal-2385 -- -- \[vpxa:setServer]

\[2007-07-07 00:07:29.796 'App' 3001264 error] \[VpxaInvtHost] Can't connect to hostd/serverd. Shutting down...

\[2007-07-07 00:07:29.796 'App' 3001264 info] \[Vpxd] Shutting down now

raoulst

[bggb29]

try to stop vpxa if it is running

To stop the ESX Server Agent - /etc/inid.d/vmware-vpxa Stop

then restart the mgmt agents

If that does not work try restarting the virtual center service on the VC server.

[raoulst]

Restarting the VC service did not help.

The vpxa process became a zombie and so i couldn't stop it. It seems, that he reason for this was a smb share mounted in the console that I used to access ISO images. Something killed the connection to the samba server, but the console had the share still mounted. Since there is no known way to kill that kind of mount I had to shutdown all VMs on that server and reboot it.

also see http://www.intrasection.com/pjmorr/2007/04/16/vmtn-discussion-forums-vmotion-without-vpxa-agent/

raoulst

[pmorrison · Accepted solution]

Wow, I'm famous.

-Phil

http://www.intrasection.com/pjmorr

[rajesh_singara1]

hi all,

i have a similar problem; my esxserver freequently loose connection with virtual center, if i reconnect then it will connect but after couple of minutes it disconnects again

please help me

thanks

rajesh

[Herschelle]

This is an old thread, you may like to start a new one.

Having said that however, you should check your dns is cofigured correctly, forward and reverse lookups. I have had this problem and it turned out to be misconfigruration in dns. If you do not use dns, then triple check all your hosts files on the esx and VC servers.

HTH