Snapshot causing VM Power Off

[eeg3]

We are running Equallogic's Auto-Snapshot Manager/VMware Edition as one part of our data protection strategy, and we have ran into an issue where every once in a blue moon we'll get an error stating:

Error message from esx-host: An error occurred while restarting virtual machine after taking a snapshot. The virtual machine will be powered off.

Error message from esx-host: Performing disk cleanup. Cannot power off.

Virtual machine on esx-host is powered off

This occurs after ASM/VE creates the snapshot then tries to remove it... resulting in the VM being powered off.

I've found a forum post (http://forums11.itrc.hp.com/service/forums/questionanswer.do?admit=109447626128381667656728353475&threadId=1099600) having similar issues with what seems to be HP's equivalent, but there didn't seem to be any solution there either.

Has anyone else ran into this issue before?

[petedr]

The VMWare hostd logs may offer some additional information ( etc/vmware/hostd.log

www.phdvirtual.com, makers of PHD Virtual Backup for Vmware and Xen Server, formally esXpress

[eeg3]

DISKLIB-VMFS : "/vmfs/volumes/4c58b856-3e066306-796c-a4badb11d718/cad-clover/cad-clover_1-flat.vmdk" : open successful (21) size = 21474836480, hd = 0. Type 3

DISKLIB-VMFS : "/vmfs/volumes/4c58b856-3e066306-796c-a4badb11d718/cad-clover/cad-clover_1-flat.vmdk" : closed.

Get null or empty snapshot tree.

Time to gather Snapshot information ( read from disk, build tree): 6 msecs.

Consolidate disks after snapshot removal..

Consolidate disks job number 11534459...

Updating current heartbeatStatus: yellow

Upgrade is required for virtual machine, version: 4

Question info: Cannot open the disk '/vmfs/volumes/4c58b856-3e066306-796c-a4badb11d718/cad-clover/cad-clover_1-000002.vmdk' or one of the snapshot disks it depends on.

Disconnect check in progress.

HandleMessage: (vim.event.VmMessageErrorEvent) {

Retrieved current VM state from foundry 4, 8

Disconnect check in progress.

Event 562 : Error message on cad-clover on est-esx4.esx.domainname.com in ha-datacenter: Reason: Operation canceled.

Auto-answered question Cannot open the disk '/vmfs/volumes/4c58b856-3e066306-796c-a4badb11d718/cad-clover/cad-clover_1-000002.vmdk' or one of the snapshot disks it depends on.

Retrieved current VM state from foundry 4, 8

Question info: An error occurred while restarting virtual machine after taking a snapshot. The virtual machine will be powered off.

HandleMessage: (vim.event.VmMessageErrorEvent) {

Disconnect check in progress.

Retrieved current VM state from foundry 4, 8

Event 563 : Error message on cad-clover on est-esx4.esx.domainname.com in ha-datacenter: An error occurred while restarting virtual machine after taking a snapshot. The virtual machine will be powered off.

Auto-answered question An error occurred while restarting virtual machine after taking a snapshot. The virtual machine will be powered off.

Disconnect check in progress.

Retrieved current VM state from foundry 4, 8

Disconnect check in progress.

Question info: Performing disk cleanup. Cannot power off.

HandleMessage: (vim.event.VmMessageErrorEvent) {

Retrieved current VM state from foundry 4, 8

Disconnect check in progress.

Event 564 : Error message on cad-clover on est-esx4.esx.domainname.com in ha-datacenter: Performing disk cleanup. Cannot power off.

Auto-answered question Performing disk cleanup. Cannot power off.

Retrieved current VM state from foundry 4, 8

Done disk consolidation after removing snapshots.

DISKLIB-VMFS : "/vmfs/volumes/4c58b856-3e066306-796c-a4badb11d718/cad-clover/cad-clover-000002-delta.vmdk" : open successful (23) size = 8589934592, hd = 0. Type 8

DISKLIB-VMFS : "/vmfs/volumes/4c58b856-3e066306-796c-a4badb11d718/cad-clover/cad-clover-000001-delta.vmdk" : open successful (23) size = 8589934592, hd = 0. Type 8

DISKLIB-VMFS : "/vmfs/volumes/4c58b856-3e066306-796c-a4badb11d718/cad-clover/cad-clover-flat.vmdk" : open successful (23) size = 8589934592, hd = 0. Type 3

DISKLIB-VMFS : "/vmfs/volumes/4c58b856-3e066306-796c-a4badb11d718/cad-clover/cad-clover-000002-delta.vmdk" : open successful (21) size = 8589934592, hd = 0. Type 8

DISKLIB-VMFS : "/vmfs/volumes/4c58b856-3e066306-796c-a4badb11d718/cad-clover/cad-clover-000002-delta.vmdk" : closed.

GetDiskParentInfo: successfully queried info /vmfs/volumes/4c58b856-3e066306-796c-a4badb11d718/cad-clover/cad-clover-000002.vmdk -> cad-clover-000001.vmdk)

DISKLIB-VMFS : "/vmfs/volumes/4c58b856-3e066306-796c-a4badb11d718/cad-clover/cad-clover-000001-delta.vmdk" : open successful (21) size = 8589934592, hd = 0. Type 8

That's the meat of the log, I believe. Full log regarding the VM in question during the time period attached. I can't really make out much from the log unfortunately, but maybe someone else can.

Thanks!

[petedr · Accepted solution]

These types of message could be the issue.

Question info: Cannot open the disk

'/vmfs/volumes/4c58b856-3e066306-796c-a4badb11d718/cad-clover/cad-clover_1-000002.vmdk'

or one of the snapshot disks it depends on.

That shows when Vmware can't find a disk in the snapshot chain. Possibly the chainhas gotten corrupt. The vmsd file would show the snapshot chain along with the vmx which shows the current disk being pointed to.

www.phdvirtual.com, makers of PHD Virtual Backup for Vmware and Xen Server, formally esXpress

[continuum]

please zip the vmx-file

cad-clover-000002.vmdk

cad-clover-000001.vmdk

cad-clover.vmdk

and all vmware*.logs together and attach them.




_________________________

VMX-parameters- WS FAQ -[ MOAcd|http://sanbarrow.com/moa241.html] - VMDK-Handbook

[petedr]

Thanks for the correct.

Were you able to resolve this issue.

www.phdvirtual.com, makers of PHD Virtual Backup for Vmware and Xen Server, formally esXpress

[eeg3]

I believe it is related to Symantec trying to snapshot and Equallogic trying to snapshot at similar times. I've switched up the schedule, and I haven't had it occur since.

Thank you for your help.