Frozen VM Client now shows as Unknown (invalid)

[whitewolfjm]

Hello,

I have a VMware ESXi 4 server with a Windows Server 2003 x64 installation that locked up yesterday. I attempted to shut the server down in the vSphere Client, but the shutdown process hung. After that I checked online for other ways to shut the server down.

I tried using the CLI and using the "stop hard" command but that did not work.

Then I went into service mode and tried both "vm-support -X" and "kill -9". Both of these processes finished successfully but when I checked the state of the clients the hung client was still listed. When I tried "kill -9" again it said that the process did not exist.

At this point I did some more research and found that if the process is killed but still shows as being active it may be related to the management agents so I restarted the management agent on the server.

Now the hung server is no longer listed as being active, however in the client the server is listed as Unknown (invalid) and as such I cannot start or manage it. I checked in the Datastore, but I cannot find any file that looks like a lock file.

Any help is greatly appreciated.

[allb18c]

I assume the VM in question is offline, correct?

If so, you will need to unregister the VM follow by a re-register of it.

It should then fire back up with no issues (assuming that none of the files are locked).

[whitewolfjm]

Ok, I was able to un-register it successfuly, however the first time I tried to register it, I recieved an error saying that it had timed out waiting for the server to respond. The server does not show up in the inventory however and so I cannot un-register it and try again.

Now when I go to re-register it I get the error message "A virtual machine

or a template having the same name is already registered. Server

error: The specified key, name or identifier already exists."

Thanks again for all the help/

[DSTAVERT]

Browse the datastore (double click the datastore) and right click on the vmx file in the Virtual Machine folder and "Add to Inventory".

[allb18c]

There may still be a lock on the files somewhere.

Without spending a whole lot of time finding which host has the lock on it, vmotion all the guests off of 1 host into another and reboot the host.

Reboot each host in the cluster and afterwards you should be finally able to register that VM guest again.

[whitewolfjm]

Yes that is what I did, and I receive this error message: "A virtual machine

or a template having the same name is already registered. Server

error: The specified key, name or identifier already exists."

[whitewolfjm]

Yikes! Well I am afraid that won't work as this is a stand alone server, and I don't have Vmotion.

So basically is sounds like the only way I can try to fix this is to wait till late this evening and then shut all the other VMs down and then reboot the whole server?

If so, that is a bit of a bummer.

Thanks for the help

[DSTAVERT]

Create a new custom VM with a new name. Don't add a disk. Once the VM is completed open the datastore browser and browse to the old folder. Right click on the vmdk an select move to. Select the new folder as the destination. Edit the settings on the new VM and add an existing disk. Browse to the new folder and select your vmdk.

[allb18c]

Ouch - that hurts.

I found this post - I dont know if it will help you or not.

http://communities.vmware.com/thread/70742

[whitewolfjm]

Before I tried this, I decided to restart the App Manger once again. This time, it re-added the server into inventory and all seemed fine until I tried to start it. Then I get this message:

Power On virtual machine | Virtual Machine | Unable to access a file since it is locked.<unspecified filename> | root | 7/17/2009 1:07:56 PM | 7/17/2009 1:07:56 PM | 7/17/2009 1:07:56 PM

So it looks like I need to reboot the server after all (Since it won't tell me what file is locked.)

If there are any other ideas, that would be awesome, but otherwise I will just have to wait till this evening to get it fixed.

Thanks again,

[DSTAVERT]

Did you try what I posted two spots up? It should take you 2 minutes to try it.

[whitewolfjm]

Hey DS,

Yeah I just tried what you recomended above by creating a new VM and moving the disk to it after it is created. I still get the unspecified file is locked error message however so it seems that I will need to reboot.

Thanks for the ideas.

[DSTAVERT]

You can try the unsupported console ALT F1 and navigate to the old folder and look for the lock files. lockfiles aren't visible in the datastore browser. From the command prompt in the folder you can use ls -a to see if the lock files exist. rm -rf (RM -RF) .lck* the DOT in front of the LCK is necessary.

[whitewolfjm]

Hey DS,

Thanks for the continuing ideas. I checked the folder for the VM and there were no .lck files listed. Looks like a reboot may be the only cure.

Thanks,

[DSTAVERT]

I don't believe in restarting servers as a way to fix problems. Last resort only. Try a find / -name '.lck*' just to see what lock files you do have.

If it is a last resort by all means and definitely getting the VM running again is first and foremost.

[johthi]

Is this VM running under cluster mode? If it is clustered, How many ESX servers running under this cluster?

Please try this out...

1. I assume the VM is running under cluster. Please check all the ESX host, if there is some jobs are running with relavant of VM name. If so please kill the process.

2. If this VM is running any one ESX server, then Login with VI client and check the VM. Still this VM show in invalid mode. unregister the VM from this ESX host. then register it via command line.

3. Try to restart the VC server service.

If this info is useful to you please mark it correct or helpful.

Regards

John