Full system hang with vmware sync / quiescing

[norrellmeister]

I've got two linux VMs, one with a modern Gentoo kernel, and one running Ubuntu 12.04 LTS. They are both displaying some nasty behavior when snapshotting or backing up. Both machines are mounting the same set of NFS shares via nfs4 from a Synology NAS. When performing a backup via VMware Data Protection, or a snapshot with filesystem quiescing enabled, both machines hang completely. They don't ever seem to un-hang without a force reset of power. Switching virtual consoles is still possible once the machines are in this state, but that's about it. Logging in or attempting to access any binaries just hangs forever.

A few finds:

1) Switching to nfsv3 resolves the issue.

2) Disabling filesystem quiescing in vmware tools via adding the following to tools.conf fixes the issue.

[vmbackup]

enableSyncDriver=false

3) Snapshots including memory, or snapshots without memory but also without filesystem quiescing, work fine in all cases.

Anyone seen this before? I've worked around this for now by disable the sync driver, but that seems like it shouldn't be needed.

4) My guess is that this related to the FIFREEZE ioctl that the vmware tools source code shows is being used to do the freeze.

[HawkieMan]

Had this kind of problem on RedHat 5.5, 6.5 and 7.0 when using OpenVM tools, but when using vmware native tools the problem disappeared.

Have also seen this problem occur on VMs having the Dell Openmanage klient compiled into the kernel. This problem was especially confusing as the VMs didnt run on any Dell hardware, but it seemed as that Dell driver was pre-compiled into the kernel.

[ravindravmw]

NFS related issue was fixed in open-vm-tools-9.4.6.

I believe you are all using open-vm-tools-9.4.0. Could you please confirm the Tools version by running "vmware-toolbox-cmd -v" command in the guest?

Also, when you say "Using VMware Tools" fixed the issue, please highlight the version of VMware Tools.

[norrellmeister]

The version I see is 9.7.0.26610 (build-1328054).

This looks like it's a later version number, but actually an earlier version.

Its compiled via Gentoo portage that appears to be downloading and using a snapshot from 2013.

Thanks for the info. I'll urge them to upgrade so they can get the fix.

[ravindravmw]

Yes, confirmed. This version does not have the fix.

This version is a development snapshot from http://sourceforge.net/projects/open-vm-tools/files/open-vm-tools/Development%20Snapshots/open-vm-to....  and it came out from our development branch, that's why it had a newer version. Ideally, this should not have been packaged for production because development snapshots are unstable and intended for developers only.