Unable to run virtual machines after the upgrade to 6.7

[Mufimufin]

Dear all,

I need your help since I struggle with virtual machines after the upgrade.

I have upgraded my ESXi 6.0 to 6.7 using SSH commands (e.g. https://miketabor.com/how-to-upgrade-esxi-6-5-to-esxi-6-7/ ). Immediately after I have signed in to the "new" web based ESXi vSphere I have noticed that:

a) virtual machines (2) have not started and have red exclamation mark

b) there are no datastores

I have tried the data store browser but web app throws JS errors on me. When I run Linux commands these are shown normally.

I have subsequently tried to downgrade ESXi, but this does not seems to be an optiion (both 6.0 and 6.5).

What I suspect is some hard drive controller driver missing (deleted during the update proces), but I do not know which one since there are many of them in the update log. Another thing that could possibly happened is that 6.0 => 6.7 was a big jump and I need to change VMFS to something new, because its no longer supported.

Update log:

https://paste.ee/p/risUi

Kernel log:

https://paste.ee/p/UZxWf

This is DYI home server:

CPU Intel Pentium G620T (2,2GHz, 3MB, LGA1155) BOX

MB MSI H61MA-E35 (B3) - H61, s.1155, DDR3, mATX

+ 3 hard drives

Thanks in advance for any help.

[Mufimufin]

Here is the detail from one of those virtual machines:

Compatibility

[missing "en-us.vm.hardwareVersion..label" translation]

VMware Tools

[missing "en-us.vm.tools.installed." translation]

[diegodco31]

In the vSphere web client

Actions > Compatibility > Upgrade VM Compatibility....

How old is this VM that it has the HW version set so low??

Here is a document talking about the hardware version upgrade: VMware Knowledge Base

Before you upgrade the virtual hardware:

1. Create a backup or snapshot of the virtual machine. For more information, see:
   
   
   • Take a Snapshot in the vSphere Web Client section in the vSphere 5.1 Virtual Machine Administration guide.
   • Take a Snapshot in the vSphere Client section in the vSphere 5.1 Virtual Machine Administration guide.
2. Upgrade VMware Tools. On Microsoft Windows virtual machines, if you upgrade the virtual hardware before you upgrade VMware Tools, the virtual machine might lose its network settings.
3. Verify that all .vmdk files are available to the ESXi/ESX hosts on a VMFS 3, VMFS 5, or NFS datastore.
4. Verify that the virtual machines are stored on VMFS 3, VMFS 5 or NFS datastores.
5. Determine the version of the virtual hardware by selecting the virtual machine from the vSphere Client or vSphere Web Client and clicking the Summary tab. The VM Version label in the Compatibility field displays the virtual hardware version

Please consider marking this answer "correct" or "helpful" if you think your question have been answered correctly.

[a_p_]

Upgrading a white-box can always be a hit, or miss.

Anyway, please follow the upgrade steps again until step 4, but replace the command with the one below, and post the output.

esxcli software profile install -d https://hostupdate.vmware.com/software/VUM/PRODUCTION/main/vmw-depot-index.xml -p ESXi-6.7.0-20180804001-standard --ok-to-remove --dry-run

Note: This command will not modify anything (--dry-run), but will show what's left over from the old installation.

André

[Mufimufin]

In the vSphere web client

Actions > Compatibility > Upgrade VM Compatibility....

Have a look at one of my screenshots above. This is unfortunately not an option, because the "Actions" button is grayed out.

How old is this VM that it has the HW version set so low??

I do not remember that. It was installed when ESXi 5.1 was the newest version, so most likely 2013. I have subsequently updated to ESXi 6.0 (through USB installation) without any problems whatsoever. I was also using the native Windows vSphere client app until today. Right now I have updated to 6.7 and problems appeared.

Here is a document talking about the hardware version upgrade: VMware Knowledge Base

Have a look at one of my screenshots above. This is unfortunately not an option, because the "Compatibility" button is grayed out.

Thank you.

[Mufimufin]

Anyway, please follow the upgrade steps again until step 4, but replace the command with the one below, and post the output.

Thanks, here is the output:

https://paste.ee/p/0ZVnN

Btw. some of those "VIBs Removed" may possibly be my test attempts that I have installed to make it work. I am just mentioning it so that we will not stuck on them while thinking that this is the actual problem (e.g. xAHCI from front.de, locker tools).

[Mufimufin]

What about vmfs not being upgraded issue? Perhaps I should do that manually using SSH command.

[root@ESXi:~] esxcli storage vmfs upgrade -l Datastore

Cannot find VMFS volume with label Datastore

[root@ESXi:~] esxcli storage core path list

sata.vmhba64-sata.0:0-naa.50014ee6567fa232

   UID: sata.vmhba64-sata.0:0-naa.50014ee6567fa232

   Runtime Name: vmhba64:C0:T0:L0

   Device: naa.50014ee6567fa232

   Device Display Name: Local ATA Disk (naa.50014ee6567fa232)

   Adapter: vmhba64

   Channel: 0

   Target: 0

   LUN: 0

   Plugin: NMP

   State: active

   Transport: sata

   Adapter Identifier: sata.vmhba64

   Target Identifier: sata.0:0

   Adapter Transport Details: Unavailable or path is unclaimed

   Target Transport Details: Unavailable or path is unclaimed

   Maximum IO Size: 32768

sata.vmhba0-sata.0:0-naa.50014ee260e467d1

   UID: sata.vmhba0-sata.0:0-naa.50014ee260e467d1

   Runtime Name: vmhba0:C0:T0:L0

   Device: naa.50014ee260e467d1

   Device Display Name: Local ATA Disk (naa.50014ee260e467d1)

   Adapter: vmhba0

   Channel: 0

   Target: 0

   LUN: 0

   Plugin: NMP

   State: active

   Transport: sata

   Adapter Identifier: sata.vmhba0

   Target Identifier: sata.0:0

   Adapter Transport Details: Unavailable or path is unclaimed

   Target Transport Details: Unavailable or path is unclaimed

   Maximum IO Size: 32768

sata.vmhba1-sata.0:0-naa.5000c5004f06c948

   UID: sata.vmhba1-sata.0:0-naa.5000c5004f06c948

   Runtime Name: vmhba1:C0:T0:L0

   Device: naa.5000c5004f06c948

   Device Display Name: Local ATA Disk (naa.5000c5004f06c948)

   Adapter: vmhba1

   Channel: 0

   Target: 0

   LUN: 0

   Plugin: NMP

   State: active

   Transport: sata

   Adapter Identifier: sata.vmhba1

   Target Identifier: sata.0:0

   Adapter Transport Details: Unavailable or path is unclaimed

   Target Transport Details: Unavailable or path is unclaimed

   Maximum IO Size: 32768

Here is the core devices list command output:

https://paste.ee/p/6ipB2

[a_p_ · Accepted solution]

There's no need to upgrade VMFS. As a side note, an upgrade from VMFS5 to VMFS6 - if you plan to do this at a later time - is no inplace upgrade. You'll have to reformat the datastores.

The vmkernel.log that you've posted contains entries for the 3 disks as being detected as sanpshot LUNs.

Please run

esxcli storage vmfs snapshot list

and if the LUNs show up, you may be able to mount the datastores from the command line by following the steps in https://kb.vmware.com/s/article/1011387

André

[Mufimufin]

here's no need to upgrade VMFS. As a side note, an upgrade from VMFS5 to VMFS6 - if you plan to do this at a later time - is no inplace upgrade. You'll have to reformat the datastores.

You got me at the reformat part, so I will put this out of my head for now.

esxcli storage vmfs snapshot list

Here is the output:

[root@ESXi:/vmfs/volumes] esxcli storage vmfs snapshot list

4fc64015-85243e40-f10f-8c89a57ce0bc

   Volume Name: datastore1

   VMFS UUID: 4fc64015-85243e40-f10f-8c89a57ce0bc

   Can mount: true

   Reason for un-mountability:

   Can resignature: true

   Reason for non-resignaturability:

   Unresolved Extent Count: 1

50eeddd9-e71fa408-61be-8c89a57ce0bc

   Volume Name: datastore2

   VMFS UUID: 50eeddd9-e71fa408-61be-8c89a57ce0bc

   Can mount: true

   Reason for un-mountability:

   Can resignature: true

   Reason for non-resignaturability:

   Unresolved Extent Count: 1

564e1a27-c0cebf32-54f2-6805ca128dcc

   Volume Name: WD Red NAS 6 Tb storage media

   VMFS UUID: 564e1a27-c0cebf32-54f2-6805ca128dcc

   Can mount: true

   Reason for un-mountability:

   Can resignature: true

   Reason for non-resignaturability:

   Unresolved Extent Count: 1

and if the LUNs show up, you may be able to mount the datastores from the command line by following the steps in https://kb.vmware.com/s/article/1011387

I had a look and tried to figure out what command should I put in there. It worked!!!!

I now have 2 of 3 datastores in my vSphere web app, but the last one did not worked.  Perhaps I should use a different syntax?

[root@ESXi:/vmfs/volumes] esxcli storage vmfs snapshot mount -l datastore1

[root@ESXi:/vmfs/volumes] esxcli storage vmfs snapshot mount -l datastore2

[root@ESXi:/vmfs/volumes] esxcli storage vmfs snapshot mount -l WD Red NAS 6 Tb storage media

Error: Invalid option Red

Usage: esxcli storage vmfs snapshot mount [cmd options]

Description:

  mount                 Mount a snapshot/replica of a VMFS volume.

Cmd options:

  -n|--no-persist       Mount the volume non-peristently; the volume will not be automounted after a restart.

  -l|--volume-label=<str>

                        The VMFS volume label of the snapshot to mount.

  -u|--volume-uuid=<str>

                        The VMFS volume uuid of the snapshot to mount.

Edit: I have added "" and even the last one worked!

Thanks for the help.

[sanjaysonwani]

Thank you very much !

Resolved my problem with provided your solution. 

root@ESX:~] esxcli storage vmfs snapshot list
61da284c-79e21b22-92ad-1866da18fae3
   Volume Name: SAMSUNG_SSD
   VMFS UUID: 61da284c-79e21b22-92ad-1866da18fae3
   Can mount: true
   Reason for un-mountability:
   Can resignature: true
   Reason for non-resignaturability:
   Unresolved Extent Count: 1

To mount a snapshot/replica LUN that is persistent across reboots, run this command:
esxcli storage vmfs snapshot mount -l label|-u uuid

[root@ESX:~]
[root@ESX:~] esxcli storage vmfs snapshot mount -l  "SAMSUNG_SSD"
[root@ESX:~] esxcli storage vmfs snapshot mount -u "61da284c-79e21b22-92ad-1866da18fae3"
No unresolved VMFS snapshots with volume UUID '61da284c-79e21b22-92ad-1866da18fae3' found

Regards

Sanjai Kumar