ESXi Allow authenticationParams=_port=8309 Can't Access UI

[jrhaakenson]

503 Service Unavailable (Failed to connect to endpoint: [N7Vmacore4Http16LocalServiceSpecE:0x000000f046c34780]_serverNamespace=/action=Allow authenticationParams=_port=8309)

One of my ESXi hosts just randomly went offline.  I cannot access the UI due to the error in the above screenshot.  I can access the DCUI and ESXi shell but I don't know where to troubleshoot.  The hostd service is not started.  If I start the hostd service using /etc/init.d/hostd start it will start but then stop again after 20-30 seconds.  Researching the error above doesn't really come up with anything specific for the Allow authenticationParams=_port=8309 portion.  I restarted the host multiple times but no resolution.  Help?

[Lalegre]

@jrhaakenson,

Could you please attach here: /var/log/hostd.log

[jrhaakenson]

I can't export the whole log at the moment but here are a couple of snippets that might provide some info.

Particularly the Failed to resolve localhost IP address, Host not found (authoritative).  Also the Unable to enable WOL capability for nic cannot be turned on probable bug file against driver ntg3

[Lalegre]

@jrhaakenson,

Seems to be showing unable to resolve but IPv6, if you are not using it, you could disable it. Also, make sure that you have A and PTR records for your ESXi.

[jrhaakenson]

I checked the DNS server and the forward and reverse lookup zones have the ESXi host entries configured correctly.  Yes we do not use IPv6 so that seems to be unrelated.  I don't think any of this is related to the hostd service failing to remain active and producing the above error when connecting to the UI.

[Lalegre]

@jrhaakenson,

I do not think too, if possible, try to start the service manually to generate fresh logs and attach the entire snippet here.

[jrhaakenson]

These are the last four pages of logs after restarting the hostd service.

[Lalegre]

@jrhaakenson,

Could you please grab the file and attach it here as a file too? Would be better to analyze.

[jrhaakenson]

Sorry but I can't get a WinSCP connection to my ESXi host to get the log file off.  Is there another way to export the log file to Windows?

[Lalegre]

If you have SSH, WinSCP should work to get it, what issue are you facing? 

You do not have vCenter right? Or your ESXi connected to it.

[jrhaakenson]

I can't seem to get SSH or WinSCP connectivity to my ESXi host at the moment so I can't use those.  I do have vCenter but the ESXi host is currently disconnected from vCenter because of the hostd service issue.  I was able to generate a vmware-support log bundle and grab the hostd.log file from there.  So it is now attached here.  Please let me know if you can find anything relating to why my hostd service will not remain started.  Thanks!

[jrhaakenson · Accepted solution]

My issue is fixed and resolved now.  It seems that a VM i had migrated from one environment to the vSAN datastore on this environment was corrupted somehow.  It may have been corrupted by some Veeam backup software running backup tasks overnight and messing up the VM file structure on the vSAN datastore but I'm not completely sure on that.  The migrated VM on the datastore was in a directory of VM_Name->VM_Name->VM files rather than just VM_Name->VM files.  There was an extra VM_Name directory on the vSAN datastore (possibly created by Veeam).  So I completely deleted the troublesome VM directory on the vSAN datastore, through vCenter (luckily this VM was unimportant), rebooted the ESXi host, and upon reboot the host services all started and the host reconnected to vCenter and the vSAN network again.  I don't really understand how a single powered off VM on the vSAN datastore could cause the ESXi host management services to fail starting up (even after multiple restarts) but that seems to be what was causing the issue.  All resolved now.

[ca_fahem]

@jrhaakenson Thank you for sharing, yesterday i had exactly the same issue with a vSAN 8 environment after migrating a VM using Veeam. i deleted the VM and restarted the management services. Thanks.