VM performance is Slow

[banumuth]

All,

One of my VM performance is very slow and Futex Process is utilizing 81.22%; I would like to know what is that and what is the solutions for this issue? In this VM I have allocated 10GB of memory and 2 CP and Please find the attached screenshot for more info.

[khughes]

Does this server need 2 CPU's? It really looks like you have over allocated resources to this VM, 10GB of ram? It is showing it isn't even using 1GB of that... What are the specs on your host? How many other VMs are running on it? Are those other VMs also multi-cpus? My guess its not a virtual machine problem it is a contension issue with the host it's sitting on.

• Kyle

[banumuth]

The hardware is DELL PE 6950 and 4Dual CPU and 64GB (15GB is currently used) memory and total 6 VM's in this Host; Yes other VM's are running into multiple CPU.

[khughes]

If you're running 4 physical dual-core CPUs, and running 6 VM's running 2-4 vCPU's you're running into CPU contention. Basically what is happening is that your VM's are waiting for cores to become available to process command sets. In your case, since they're running multiple vCPUs they have to wait for 2 cores to become available before it can process a single instruction set. Virtual environments are different than physical environments when it comes to allocating resources. In a physical environment you could just load up a server with CPU's and RAM and it wouldn't think anything different, but in a virtual environment less is usually more. Best practice is to assign only 1 vCPU to start and if you find it needs the second then assign it. I would say about 90% of all old physical dual CPU machines run better as a 1 vCPU VM over a 2vCPU VM.

• Kyle

[banumuth]

From ESXTOP I am seeing average 40% (41% inVCPU1 and 39% Vcpu2 )CPU only used in this VM and 921MB out of 10GB used. but still app team saying some performance issue is there. Please let me know if you have any other thought....

[khughes]

percentages are one thing, what I'm looking for is the ready% column. what are those numbers within esxtop?

• Kyle

[banumuth]

Hello ,

As I said I have reserved 10GB memory for this VM and Its saying usage as per my previoues attachement, but the Linux admin are showing its utilizing 7.1GB out of that 10GB What is the reason why the VI Client shows as attached?ESXTOP showing used memory 1GB but Linux kernel showing usage 7.1GB ? Why?

[mcowger]

Bewcause the Linux kernel reports memory used as cache as 'in use' when its not really. What does 'free -m' show?

Also, if the VM is waiting on a futex lock, thats not indicative of a problem with the VM, thats indicative of a problem with the app or OS.

--Matt

[banumuth]

yes futex utilizing 81.2% Why?

*

free -t -m

total used free shared buffers cached

Mem: 9894 8126 1768 0 348 5329

-/+ buffers/cache: 2448 7446

Swap: 10268 0 10268

Total: 20163 8126 12036

*

[mcowger]

According to the output of free there - your VM really has 7.4GB free.

As far as the futex lock - thats up to your application, not VMware - you'd have to ask the application developer / OEM why its app is waiting on futex locks.

--Matt

[banumuth]

CPU %rdy 5.50? and These is the only one VM avaiable VM in that ESX.

[tcutts]

We've had futex problems ourselves, in particular with java applications.

I'm pretty certain, at the moment, that the problem lies in the way the Linux guest measures time. On the guest, check the output of:

cat /sys/devices/system/clocksource/clocksource0/current_clocksource

It it's set to 'tsc' or 'jiffies' you're likely to run into serious timing problems on the VM. Try running:

echo acpi_pm > /sys/devices/system/clocksource/clocksource0/current_clocksource

and see if your timing worries go away. Note, if your machine is running NTP, you probably will need to stop NTP, synchronise the time manually, and then restart NTP.

Using acpi_pm as the clocksource seems to have done the trick for us, although it's early days. To make the above change permanent, append clocksource=acpi_pm to the kernel options in /boot/grub/menu.lst

You could also try the vmware-toolbox option of getting VMware itself to set synchronise the virtual machine's time. I have not yet tried that option.

Tim

[tcutts]

False hope... While using the acpi_pm clocksource initially seemed to work well, the problem has returned.

[tcutts]

I noticed that the problems we're seeing are all common to Debian Lenny guests, and we haven't seen these problems with Debian Etch. One of the changes from Etch to Lenny was the use of a tickless kernel. I'm building a kernel now without tickless support, and with CONFIG_HZ reduced to 100Hz, and will try that on a Lenny guest to see if the performance of the application improves.

[softlion]

yep, same problem here.

Host is debian lenny x64 /4Gb ram without tickless option running vmware server 2.0.1 and two guests: server 2003 x86 (2Gb ram) and server 2008 x64 (1Gb ram).

The current_clocksource option made things a bit faster.

I've experienced very slow startup on VMs sometimes. I can't exactly reproduce it but when it's slow the only choice is a host reboot.Stopping/restarting service doesn't do anything.

Clock speeds displayed in web access are ... funny.

And overall performance is still not as good as I would like it to be.

There is no option to set vm clock speeds ?

Also I get a lot of ServiceNotAvailableException from web access. An idea ?

SL.

[tcutts]

yep, same problem here.

Host is debian lenny x64 /4Gb ram without tickless option running vmware server 2.0.1 and two guests: server 2003 x86 (2Gb ram) and server 2008 x64 (1Gb ram).

Ah, no, that's not quite the same problem as me (although it's interesting in itself!). In my case, Debian Lenny is the guest, not the host. The host is ESX 3.5u2.

Tim

[softlion]

Oh sorry I thought I've read throughfully.

I found an optimization :

echo cfq > /sys/block/sda/queue/scheduler

echo cfq > /sys/block/sdb/queue/scheduler

Check :

cat /sys/block/sd*/queue/scheduler

noop anticipatory deadline

noop anticipatory deadline

before it was in "deadline" mode:

noop anticipatory cfq

noop anticipatory cfq