Poor performance cloning a powered on VM

[NPBrown]

I'm investigating an issue on a ESX 6 (6.0.0, 3073146) cluster where cloning of powered on VM is extremely slow. Using ESXTOP I have been able to establish the following:

- Cloning a powered off 60GB VM takes around 2 minutes and uses VAAI (verified by watching CLONE_RD & CLONE_WR counters increase, whilst MBREAD/s MBWRTN/s remain low)

- Cloning a powered on 60GB VM takes around 60 minutes and doesn't use seem to use VAAI. CLONE_RD & CLONE_WR counters remain static. MBREAD/s MBWRTN/s settle at around 17 during the process after an initial burst of around 250.

- Using VMKFSTOOLS to copy a 60GB disk takes around 2 minutes with VAAI enabled on the host (again verified using CLONE_RD & CLONE_WR, MBREAD/s & MBWRTN/s counters)

- Using VMKFSTOOLS to copy a 60GB disk takes around 3 minutes with VAAI disabled on the host. MBREAD/s MBWRTN/s settle at around 350 during the process.

This begs two questions:

- Why is VAAI not being used when the VM is powered on?

- Why is the clone transfer rate so poor even when VAAI is not being used, given that VMKFSTOOLS performance without VAAI is good?

The cluster comprises four Dell R820s, 16GB Emulex HBAs, Dell Compellent SAN. ESXi is installed onto onboard SD card with scratch path redirected to a persistent datastore. Datastores are clean VMFS5 and storage I/O control is not enabled. I have the SAN team looking at any potential compatibility settings on the back end, but in the meantime any suggestions for further diagnosis much appreciated.

Cheers, Nick.

[jonathanp]

The only difference with power on vs power off vms cloning process is that a snapshot is created...

So, depending on what is going on, on a powered on VMs, the snapshot operation ( creation / removal ) may takes longer times.

Same way as when you do a snapshot on a powered off vms vs a snapshot on a powered on vms.

So, I would try if you have approximately the same duration ( probably a little less long ) if you just try do to a snapshot on your VMs powered on and remove it...

If it takes long, then, you can looks at different VMware KB about slow snapshot operation, there are plenty KBs related to that out there.

Also if there are applications running on the VMs, like database, etc, you could try to pause these apps during the clone operation.

We sometimes have issue with VMs, and consolidation takes forever.. and after cloning it, these operation goes way faster on the cloned VMs..

Also, a note from VMware KB: Live clones can be expected to result in a slightly out-of-date destination virtual machine, where data recorded (to snapshot delta disks, during the clone operation) is not committed to the destination copy.

Hope this helps a little.

Jon

[NPBrown]

Hi Jon, thanks for the response. So while snapshot creation isn't the issue - it only takes a few seconds, this is a very quiet system I'm testing on - it did get me round to thinking of another test I could do. I created a snapshot, powered the VM off and performed the clone. Sure enough the clone operation performed as poorly as peviously when attempting to clone the VM powered-on. So now I've got the powered off clone performing like the powered on clone. I'm seeing the same symptoms - no VAAI being used, low MBREAD/s MBWRTN/s numbers and 55 minutes to complete. If I remove the snapshot from the powered off system the clone is completed in 2 minutes once more.

So it appears to be the presence of the snapshot that is the problem - but still don't know why.