Virtual machine not starting after upgrade to 5.1

[RemkoDeKoning]

Hi there. I have a virtual machine that won't start up after an upgrade from ESXi 4 --> 5.1

All other VM's work fine, just this one.

It looks like the VM profile is damaged of some sort. I cannot import the VM into the inventory.

When I add my .vmx to the inventory I get "unknown" vm

I have tried to create a new VM and simply importing the Virtual Disk but then my Snapshots are not showing.

What can I do to sort this out?

I really really need the snapshots 🙂

Remko

[a_nut_in]

Hey Remko,

How many snapshots do you have? Open an ssh session to the host and browse to the datastore.

In the VM folder, an ls -lrth will show you the files

open the *.vmx and check which *.vmdk file the vmx file actually has.

Create a new VM and point it to the vmdk file of the VM you are having issues with

Save the changes but do not power on the VM

Next open the vmx file using vi or any other editor and replace the *.vmdk file with the *vmdk snapshot file as mentioned in the original vmx file.

For example, lets say I have a VM with one disk and one snapshot and the hdd is called test.vmdk

There would be a test-000001.vmdk should there be a snapshot

In the VMX file, the disk would point to test-00001.vmdk

Create a new VM - (you will get a vmx file) and point it to test.vmdk

Next ediot the file and manually point it back to test-00001.vmdk

Unregister and register the vmx file to the inventory and power on.

An alternative approach would be to first commit the snapshot file.

To do that, run the following command:

vmkfstools -i test-00001.vmdk new_file_name.vmdk

Once the clone is done, point the new_file_name.vmdk to the newly created vmx file and power on

[RemkoDeKoning]

Hey, thanks for this elaborate answer. Much appreciated!

I did check a  couple of things. Looks like the original VMX file is just empty. There is nothing in it.

I checked other Virtual Machines and they all have VMX files with all sorts of configuration data in them.

About your other remark not to start up the VM. When we rebuild the VM and attached the Virtual Disks we discovered (after startup) that it contained old data. The rest is most likely in the snapshots.

Does this mean we are "screwed" ???:smileycry::smileycry:

[a_nut_in]

Hmm.

Try this:

The VMDK is actually a collection of two files a vmdk and a flat

Open a PUTTY to the host and lets say snapshot 00004 is the last disk in the chain, manually change the CID/PID and the "parent" file name so that the chain looks like this:

Last File

CID 123456

PID 67890

Parent test000003.vmdk

ensure you rebuild the chain so that the chain looks like this

Second Last file (000003.vmdk)

CID 67890

PID depends on the next CID

Parent again depends - but most likely 000002.vmdk

The last file should point back to the base disk, i.e. test.vmdk and the PID for that file will be all F's i.e. ffffff (depending on how many characters your snapshot chain contains)

Then keeping the parent files intact do a vmkfstools -i "last file name" (e.g 000004.vmdk) "new name"(e.g test.vmdk)

So something like

vmkfstools -i test-000004.vmdk new_test.vmdk

This will consolidate the snapshots to a single file. Point the new file to the VMX and power on.

http://kb.vmware.com/kb/1007969

[RemkoDeKoning]

Wow, it took me a time to comprehend what has been said.

To be honest, I am a bit of a novice with VMWare and just managed to learn my way around with the vSphere Client.

Working via the command line is something completely different but I can now move around a bit and list folder contents using Putty.

I did notice that more files are being showed using the CLI and the web datastore browser then with the datastore browser on the vSphere Client. This is why I initially did not comprehend the whole thing with CID’s and ParentCID’s. I simply did not see the files.

I did manage to find the snapshot files and the original flat file but there is absolutely no relationship according to the CID and ParentCID numbers.

I guess I can now try to rebuild the relationships by editing the .VMX files but am unsure how to upload the edited .VMX files. I suppose I could use the upload feature of the datastore browser of the vSphere Client but haven’t found the time yet to test this.

Thanks all for being so patient with me and for the excellent suggestions. This is much appreciated.

Remko

[continuum]

The vmx-file can be recreated by reading old logs - if you post the old vmware.logs recreating the vmx is a matter of seconds.

Old vmware.logs will also give you the correct sequence for rebuilding the snapshot-chain.
The snapshot with the highest number is not necessarily the latest one

[RemkoDeKoning]

Hey continuum. What a great offer!

I have attached all logfiles I could find in the folder. I would think, bij looking at the files, that I am missing some.

Unfortunately these are the only ones I could find.

Would this work for you?

I am a bit flabbergasted to be honest that this is even possbile. Please let me know if you need anything else.

Remko

[FedericoAlves]

Dear Continuum

I booted my ESX with your recovery ISO, and I can see my VMFS partition with parted (it is a GPT disk).

So ideally I would mount my partition and scp my VMDKs out. How can I do that?

I cannot find a manual for your recovery ISO.

[RemkoDeKoning]

Hi Federico, are sure sure you are replying to the correct topic?

[continuum]

Remko - there are several possible vmx-files that can be created using the logs

Only you can know which one makes most sense.

I attach the tool that I use to create the vmx-files.
Just doubleclick it - and select that vmware*.log that by inspecting the date - makes the most sense.
The tool then creates a vmx-file and opens it in notepad so that you safe it as the name you want

[RemkoDeKoning]

The whole concept is clear now. Learned a lot the last few days.

Unfortunately the suggestions did not work. I believe this is due to our own mistake. We have powered on the machine without the snapshots and therefore altering the main harddisk file.

A good learning experience.

Luckily, this was a testing environment for one of our production systems. My colleague had just written some extra routines that have been lost now.

A good learning lesson. Having backups available for the production system but never implemented a backup solution for the test environment.

Losing a machine is one of the nightmares of a SysAdmin. It just happened to us ☹.

Luckily no production data is lost. Just a couple of days of work in a test environment.

We will start rebuilding the test environment (also a very good lesson ☺ ) and make sure that we have backups in place from now on.

I would like to thank everyone involved in thinking of a solution. Your help is very much appreciated!!!!

Remko

[continuum]

Changing the basedisk does not kill the VM.

Read my notes http://sanbarrow.com/vmdk-handbook.html - that will prepare you for the next time.

If you power on a VM that only uses the basedisk instead of the full snapshot chain you will notice an outdated state of the VM.
When you see that stop the VM immediatly and do not run checkdisk or defragmentation tools.

Most of the times a VM will survive that - especially if the VM uses NTFS and runs Windows