Very slow, constant disc access

[Cactus_Jack]

Running VMWare Workstation v6.5.0 Build-118166, on Windows XP SP3 host (Dell Optiplex 755). VM sessions are stored on internal 500GB SATA 3GB drive. After starting Windows XP SP3 in a VM session, system response seems very slow; ie, clicking an icon takes several seconds to respond, launched apps take unusually long to load, etc. The symptoms persist for several minutes after the XP session has started. During this time, the disc indicator in the VMWare Workstation status bar shows constant disc access.

I have defraged the session, both within Windows and using the VMware Workstation defrag utility.

Anyone have any ideas why this disc access continues for so long?

Thanks,

Jack

[Cactus_Jack]

Yes, snapshots taken with power off, for the very reason you mention.

"Take and restore snapshots in the background" is already disabled.

I often run two guests at a time; however, may not be actively running applications on both at the same time. In addition, the slowness is evident even with only one session powered on.

Our Antivirus is centrally managed, with user access to it's configuration disabled.

Thanks,

Jack

[Scissor]

Does the slowness in a Guest happen only the first time you start it during a single day? That's the behavior I seem to get from the crappy Symantec AntiVirus client installed in some of my Guests... If I reboot the Guest the slowness doesn't happen again until the next day. (unless I revert to an old snapshot).

[Cactus_Jack]

No, it happens every time I start a guest.

[Scissor]

To rule out snapshots as the cause of the slowness, you could try temporarily making a Full Clone of your existing Guest. Then boot the Clone to see if it is slow at startup as well. If it is still slow then snapshots are not the cause. Delete the Full Clone after the test is done so that you don't end up with multiple copies of the "same" windows installation.

=====

Just curious -- start perfmon on your Host and then boot your Guest. While the Guest is showing constant disk access, what does the "Avg Disk Queue Length" parameter show? If it stays high (the exact number depends on the actual disk subsystem, but I would guess 3) for extended periods of time that would seem to confirm Disk I/O as the bottleneck.

[Scissor]

Maybe 512 MB RAM isn't enough for your Guest? Try bumping the amount of allocated RAM a little higher to see if that eases the startup time.

[Cactus_Jack]

Tried the full clone. same symptoms. (In fact, if anything, it was even slower.)

Upped guest memory to 1GB. Seemed ever so slightly better (wishful thinking?).

Ran Perfomon, as suggested. See attached files for results. Perfmon1 was the time from the login prompt for the first 1 min 40 secs (the default; couldn't find where to increase that). Perfmon2 was the same time period after the Windows desktop completed loading.

Here's something interesting that may help: I booted the Guest to the login prompt, intending to start Perfmon when I hit OK on login. Before I could do that, I was distracted for maybe 15 minutes. Got back to the Guest and logged in, and noted that system performance seemed to be normal! No solid disk access indicator. This means that whatever is slowing the system, it's not related to things that load AFTER login.

Jack

[Scissor]

The perfmon charts don't look too bad.

=======

I'm not sure how your IT department sets up system images, but I wonder if additional software is installed via Group Policy or other methods after the initial system installation is complete? That might explain why boot times are quick at first but then bog down after a few reboots...

Or perhaps whatever software you are testing changes something to cause an increase in boot time. Does your testing require the installation of any database software or other Services that could increase the time it takes to start up?

I still have a hunch about the AntiVirus installed in your Guest. Try (temporarily) disabling the AV service(s) in your Guest and reboot it to see if that makes any difference.

[Cactus_Jack]

Actually, it's my department within IT that sets up all machine images for the firm, so I'm very familiar with what gets installed / updated / etc. There's nothing being added via Group Policy. And the purpose of these VM guests is to test updates / installs that are pushed out via our in-house software management tool. When I build a new guest, it is based on a very recent base image that has all of the major core applications already installed, and it will have all updates pushed within one or two system starts. No database apps are installed.

Many of the updates that I test are deployed to all machines (35,000 +) in the firm; some are "individualized" and would only go to certain sub-sets of machines. If we were deploying something that causes system performance to bog down so badly for so long, I've got to believe we would have heard about it. (Depending on staff level, some users are VERY GOOD at pointing out issues....)

As I noted above, if I do not respond to the login prompt, I observe the VM disk indicator on solid for a few minutes. If I wait for disk activity to stop and then log in, performance is normal. No software updates, if any are needed, are pushed before the user logs in. This also seems to rule out AntiVirus as a cause, although I too still question that. I'll be able to get an override, allowing me to disable real-time virus scanning to test that more.

I really appreciate the efforts to help me find a solution to this annoyance; I don't want to seem like a pest.

Jack

[Scissor]

I really appreciate the efforts to help me find a solution to this annoyance; I don't want to seem like a pest.

Hahah don't worry about being a pest. This is now an interesting problem, especially since we have almost ruled out VMware Workstation as the cause.

Since virtualized disk I/O is relatively slow, Guests often "notice" high disk utilization before it is obvious on physical hardware. (You can alleviate the issue a bit by preallocating your Guest's disks, but the space tradeoff often isn't worth it)

I'm really looking forward to finding out what the problem is.

[HakanAndersson]

Finally I found somebody that has exactly my problem. I am hoping on some progress on this area. I have a development server that takes around 30 minutes before the disk access suddenly stops and suddenly the machine is like any other of my VMWare machines. This is not a hardware problem since I am running several virtual machines with the same configuration on the same disks without any problems.

There is a couple of things regarding this issue that makes it hard to troubleshoot.

• One strange thing is that there is nothing out of the ordinary when looking in Process Monitor or Process Explorer. It seems like the disk-access is hidden in some way i.e. not owned by any process. The disk-queue is however full all the time.

• The heavy IO starts very early (prior to login screen is visible) so if for instance doing a CheckDsk, which is scheduled to be peformed during startup, this is affected by the same issue making the CheckDsk take forever.

• If starting in safe-mode the same thing happends and I more or less must wait in 20-30 minutes before I can work on the machine.

• When checking perfmon on the host it is the vmware-vmx.exe that is causing the operations. The overwhelming majority of the requests are write requests.

• This happends on each reboot.

I previously had this machine on an ESX server where I had the same problem and thought that by moving it to vmware-workstation (with vmware converter) could fix the problem. I also tried to remake the disks in all the different flavors.

My guess is that it is a driver that is malfunctioning. I guess that this access would be hidden in some way. Could it be a virtual BIOS issue/config?

Best regards

Hakan

[HakanAndersson]

The machine is now as fast as usual and here are some more clues to the problem.

The virtual machine contains two disks so I moved the system-volume to I: and the data-volume to J: and kept the vmx and other associated files on H:. Now the files reside on three different physical disks. The only files still remaining on H: is the following files: <machine>.vmx (2kb + lck-file), <machine>.vmsd (1kb), <machine>-.nvram (9kb) and a <guid>.vmem file (2gb + lck-file).

Looking in perfmon on the host there is a big queue of operations towards the H:-drive and the H: disk is unresponsive to the extent that working towards it (copying files, etc.) takes forever. The virtual machine is however now usable since there is no operations going on towards the physical disks that contain the virtual volumes.

I am not really sure what this tells me more than that there is no problem with the Windows Server 2003 OS (what I can see). My guess is that all operations are performed towards the <guid>.vmem file but when looking in filemon on the host I can not see any operations towards the H:-drive at all. This is strange since I can see and hear that the disk is working really hard. Any tips would be helpful.

Best regards

Hakan

Message was edited by: HakanAndersson