After a reboot of a server, it can happen that it is not possible to take a quiesced snapshot.It happened after an update to 6U1 from 6.0. Before the update there were no problems.
We did the following to troubleshoot the problem.
* Reinstall VMWare Tools
* Check VSS Providers for errors (no errors)
What we found out was that it can happen after a reboot of the server and only if the service "VMware Snapshot Provider" is running (service is set to manual). To correct it we need to stop "VMware Snapshot Provider" and "VMware Tools". If all is stable, ie no running services, which should not run, like "Volume Shadow Copy" we start the service "VMware Tools". After this we can take a quiesced snapshot again.
While this ain't a huge problem, it is annoying at best. Any one have a solution or have the same problem.
Hi,
IF your VM's OS is Windows 2008 R2, please read the below KB and do the mentioned instruction:
VMware KB: Cannot take a quiesced snapshot of Windows 2008 R2 virtual machine
Thanks for your reply and I have read this article several times when in search for a solution.
Unfortunately this is not the solution as our backup relies on application awareness (SQL, Exchange etc.). Disabling this feature will have impact on our backup process.
All our servers (w2k8 and w2k12) have the same problem. After a reboot, the service "vmware service provider" keeps running and wont stop (not always but it can happen), which blocks quiesced snapshot.
Hi Shoomy,
I can confirm your observations.
We also experience the issue with some of our Windows 2012 servers and stopping the 'VMware Snapshot Provider' service resolves the issue.
After stopping the service it is possible again to create a quiesced snapshot of the server.
Thanks!
what is the startup type for the service ? Manual or Automatic?
I believe it should be manual.
Startup for the 'VMware Snapshot Provider' service is indeed set to manual.
Strange thing is that the issue is completely random and so far only a couple of servers (all Win2K12) are affected.
What happens is that after a reboot VMware tools service starts and triggers "VMware Snapshot Provider" and "Volume Shadow Copy" after a while, few minutes at the most, both services should stop running. Both service are on manual as VMware best practice. Exactly the same behaviour as when taking an quiesced snapshot.
In some cases "VMware snapshot provider" just does not stop and this only happens after a reboot.
Thanks Odin for confirming. May I ask you which version of VMware you are running. We are on 6.0U1 (10240) and yes it is completely random. I have it on w2k8 and w2k12, not only on w2k12.
Hi Shoomy,
So far i have only encountered the issue with a couple of Win2K12 servers.
Still trying to figure out why this occurs...
Reinstall of the VMware Tools does not solve the issue as you already stated in an earlier post.
Build : Releasebuild-3380124
Patch : 26
Update : 1
Version : 6.0.0
VMware Tools: 10240
Hey i'm having the same problem with Server 2012 R2 backups (Veeam 9 Update 1, using VeeamZIP full method). I think the problem started around ESXi 6 Update 1 or Update 1b, I'm running ESXi 6 Update 2 now.
I have reinstalled VMWare tools (with setup /C), and upgraded to ESXI 6u2 but it didn't help.
I get this in the vSphere client:
An error occurred while taking a snapshot: msg.snapshot.error-QUIESCINGERROR.
An error occurred while saving the snapshot: msg.snapshot.error-QUIESCINGERROR.
or
A timeout occurred while communicating with VMware Tools in the virtual machine.
The only error I get on the Windows guest is:
Event ID 7011 Service control manager
A timeout (30000 milliseconds) was reached while waiting for a transaction response from the VMTools service.
No VSS or any other errors.
As of now the only fix I have is to stop the VMWare Snapshot Provider service before I run a backup. This seems to work 100% for me.
The VMWare Snapshot Provider seems to starts and stops with the VMware tools service, it is set to manual (default setting) start but never seems to stop on it's own.
@a11112f
Same Problem here with ESXi 5.5 U3b!! We have >1000 VM's and a day certainly 2-3 VM's with this problem :smileyconfused:
I run a Task Scheduled Powershell script targeted at Veeam for my backups so I was able to add some lines to stop the VMware Snapshot Provider service when the backup starts. For the first time my backups all completed.
$ServiceStat = Get-Service -Name vmvss -ComputerName $VMName
if ($ServiceStat.Status -eq "Running"){
stop-service -inputobject $(get-service -ComputerName $VMName -Name vmvss)
}
I was also thinking of using Group Policy with a WMI filter targeted at Server 2012 R2 installs to stop the services, but I have the feeling that having the service stop during a backup would be...bad. Once that service is stopped though, quiesced snapshots work.
I attached the full Veeam script if anyone wants it, I didn't write the original but I have made a few of my own changes. There is more info in the script comments as to the author.
I am seeing the exact same thing under Win 2008 R2 and Win 2012 R2. We are running ESXi 6.0 U2. The VMWare Snapshot provider service is set to manual but upon reboot of the VM the service starts and does not stop. At backup time a quiesced snapshot cannot be taken. If I go in and stop this service, then a snapshot can be taken. We are using Commvault for backup but will be jumping to Veeam in the near future so I will test the script
We are seeing the same thing on all version of the 10.x tools, even the 10.0.8 version. They say this will be resolved in 10.0.9 released in June, but I am pushing for a faster resolution. I'm also trying to get them to give me some details on what exact tools / OS versions are affected and what is causing the issue.
I have 41 Windows VMs running under ESXI 6 Update 2
Yesterday I applied 2 Windows updates to all of them, they all had to restart
Last night my backup ran and it shows this morning that for 13 of the VMs a quiesced snapshot could not be taken
I restarted all 13 VMs this morning
I logged into each one and found the VMware Snapshot Provider service still running on 6 of them even though it is set for manual. I stopped this service on all 6
My backup should run fine tonight
The big fix for this cannot come soon enough
Do you by chance run VMTurbo?
Having the same issue here with 5.5 and 6U2 - we have several environments that share a base template, and since the tools have moved to 10.0.x we've been having this issue across all of them. We also use NetBackup which uses the snapshot functionality, so this meant no backups for us too.
I've found two ways around it:
1. Remove the shadow copy (VSS driver) from the VMware Tools. Bare in mind that you are removing the quiescing abilities so choose this option carefully where it works in tandem with the application and how it uses the filesystem.
2. Restart the services just after startup. I've implemented a script via gpo to roll to any affected machines - not ideal but this way it'll be easy to pull it back out later if/when the bug is fixed.
Here's the code if it helps anyone out:
sleep 60 #give the system time to start the services - may need tweaking for slower systems
if ((Get-Service -Name vmvss).Status -eq 'Running'){stop-service -inputobject $(get-service -Name vmvss)} #stop the VMware Snapshot Provider service
if ((Get-Service -Name VMTools).Status -eq 'Running'){stop-service -inputobject $(get-service -Name VMTools)} #stop the VMware Tools service
Sleep 10 #wait for the services to stop - may need tweaking but this works for us - i was too lazy to write a check loop
Start-Service -InputObject (Get-Service -name VMTools) #start VMware Tools service
The 10.0.9 tools were released last week and the notes say this issue should be fixed.
Can anyone confirm whether this is fixed with VMware Tools 10.0.9?
We have the same problem here trying to take backup using Acronis.
Several 2012 R2 server with no problem except for just one SQL Server.
The problem started suddenly maybe after a windows update (not sure) but there are other SQL Server which get updated every week.
Disabling vmware snapshot provider service works for us but don't know if there are any drawbacks.
Updated to tools v10.0.9 build 3917699 did NOT resolve the issue.
Tried also to update ESX host to last patch 6.0.0 4192238 and did NOT resolve the issue.
Problem arise only when take snapshot quiesce=yes memory=no with vmware snapshot provider service not set to disabled.
I can confirm this for latest ESXi 5.5 4345813 and Server 2003 32bit vm. When i take Snapshot, i get : msg.snapshot.error-QUIESCINGERROR.
But is worked before for some time. And also 2 other 2003 VM's dont show this Problem?! Downgrade to Tools 9.4.15 helped.