Hyperthreading On but 8 out of 16 CPUs are unused (2%-5% CPU usage) while Average CPU Ready is high

[RobinVm1]

Hi,

i would be happy if someone could clear up things a bit on this subject.

We have a DL 380 G6 running esxi4u1, Hyperthreading on and therefore 16 Logical Processors.

When checking the CPU-Diagram i see, that processors 0-7 each have a usage for about 30%-50% and processors 8-15 usage is about 2%-5%

At the same time the "average cpu ready" shows about 5-20%.

as far as i understand "hight average cpu ready" means that there are not enough cpu's ready to work for that moment.

How does this fit to the fact that there are 8 cpu's unused?

Is there anything that can be done to get lower "average cpu ready" values and using the 8 unused CPU's?

regards

Robin Seeländer

[weinstein5]

Welcome to the Forums - Remember a virtual cpu runs on a single logical cpu at a time - so I am betting you do not have many vcpus running on this box- average cpy ready % is good but should also be compared to the CPU ready time - what is the ready time you are experiencing - I will bet that it is low

If you find this or any other answer useful please consider awarding points by marking the answer correct or helpful

[weinstein5]

Welcome to the Forums - Remember a virtual cpu runs on a single logical cpu at a time - so I am betting you do not have many vcpus running on this box- average cpy ready % is good but should also be compared to the CPU ready time - what is the ready time you are experiencing - I will bet that it is low

If you find this or any other answer useful please consider awarding points by marking the answer correct or helpful

[RobinVm1]

Thanks for that quick answer.

Possibly there is a problem in wording here. I use Veeam-Monitor to monitor the esxi.

In Veeam i use "Select counters" and choose "CPU Ready" but the diagram shows "average CPU Ready".

In the VSphere Client --> Performance tab -->Chart Options --> CPU : I only see "CPU used", "CPU idle", "Reserved Capacity", "Cpu usage in MHz" and "CPU usage". There is no "CPU ready". So i don't know how i can find the value you would like to know.

The counter "CPU idle" shows an average of 5'000 for CPU 0-7 and an average of 10'000 for CPU 8-15

The counter "CPU used" shows an average of 3'000 to 9'000 for CPU 0-7 and an

average of 0-500 for CPU 8-15

You are right. I do have more vcpus than logical CPUs but still i do not see, why the CPU 8-15 are practically unused, while the value for "average cpu ready" starts to rise...

What can i do to get these 8 cpu's used? I mean, if CPU 0-7 was 100% used i could understand, that 8-15 cannot get used because these are only hyperthreaded CPUs but as long as the CPU's 0-7 have reserves they could make use of the hyperthreading feature... i am misunderstanding something here?

[weinstein5]

CPU Ready is a VM metric - you will have to be on the performance charts for the VM to see VM ready -

If you find this or any other answer useful please consider awarding points by marking the answer correct or helpful

[J1mbo]

Re hyperthreading - bear in mind it is a technology that enables greater overall utilisation of each CPU core by enabling running of another thread whilst the primary thread is stalled by (for example) a processor cache miss (and hence memory fetch). The 16 CPUs ESX reports are not really proper CPUs, of which there are 8 in this case. Hence the box will never be able to actually run 16x clock-speed worth of actual work.

FWIW on two of my (identical) boxes, I find logical CPUs 1..7 with low activity on one, and 8..15 low on the other. Hence I guess it just depends how ESX is scheduling things at the time.

Please award points to any useful answer.

[RobinVm1]

hi J1mbo,

well i am aware of the fact, that we only have 8 real cpu's in that system. And the hyperthreaded CPU's only can do work if the real CPU lying underneath is not too busy...But if the "Average CPU ready" rises, which means that there are not enough CPU's ready while 8 of them are practically unused and the others are only used 30%-50% , for me this looks like there could be some processing reserves that are unused and should be used by the hyperthreaded CPU's.

Are there any switches that can be used to optimize the usage of the hyperthreading feature?

[RobinVm1]

Well the CPU ready for each vm is between 500 and 2000. For one vm there was a peak of 9000.

As far as it looks, the hyperthreading feature is not really used. is is better to switch it off then in the bios?

[J1mbo]

Perhaps you could provide some info on the running VMs, and in particular the use of vSMP on them.

Please award points to any useful answer.

[RobinVm1]

There are 10 VMs running on that host.

All are Windows-XP machines with ServiePack3.

They have between 2 and 4 vCPUs.

They are used for Compiling and testing software. (VisualStudio, Java, Junit)

While the VMs are active and compile, the Host's CPUs (0-7) are used at a maximum of 50% (while the "hyperthreaded CPUs" 8-15 use about 1%-5%)

[J1mbo]

There are 10 VMs running on that host...They have between 2 and 4 vCPUs

My guess is that this overcommittment of vSMP is the cause of your issue, if in fact there is one. If you have two 4 vCPU guests on an 8-core server, the two cannot run concurrently for any length of time since ESX will always jump-in on core 0 for system tasks (storage, vswitch etc), let alone other guests.

Please award points to any useful answer.

[RobinVm1]

Well of course we overcommit vSMP, thats one reason we use esx-server- to utilize the 8(16) CPU's more effectively.

Still i do not see any reason, why half of the logical CPU's are unused although the other half are not fully in use... Remember half of the CPU's are running only on 30-50%...

What do you suggest?

Is it better to switch off hyperthreading in BIOS, since these CPU's are not in use anyway?

Any setting that let's ESXi make better use of the hyperthreaded CPUs?

Use a maximum of 2 vCPUs per VM?

[J1mbo]

Using 1 vCPU per guest (and 2 vCPU on an exception basis) would be a good place to start, which will enable higher overall utilisation of the hardware. This issue is essentially that for a vSMP guest, any use of the idle thread blots-out the core it's running on from anything else.

Assuming the servers have Nehalem CPUs (Intel 55xx), leaving hyperthreading enabled is recommended, even for historically picky apps like Oracle.

[RobinVm1]

Thanks, i will try using 1-2 vCPU's, although we have seen that compiling on a guest with 4 vCPUs has a positive effect on the compiling performance...

Yes we have Nehalem CPUs (Intel 55xx). if i find time to try it, i will have a look on the CPU usage, while Hyperthreading is disabled, just as a try. But i understand, that is recommended to have Hyperthreading enabled even if it is unused...

[J1mbo]

IMO it's not so much that HT is not used (much), but rather that it doesn't really do that much in the first place - c.10% is a figure sometimes mentioned.

Please award points to any useful answer.

[Callifo]

Is this not due to the fact that if you have 4vCPU VMs even if the host inside the VM only uses 1 CPU, on the outside the hypervisor must schedule all 4 vCPUs even if 3 of them simply run IDLE processes. All 4 vCPUs must be scheduled at the sametime and can cause it to starve other VMs or can starve itself of CPU time (if there are lots of single and dual vCPU VMs running busily). This was how it was described on at the VM course.

This means that it is likely that the multi processor VMs are hogging all the CPU time and could be causing your high CPU ready times on some of the virtual machines?

Said course said that you should not use mutli CPU VMs and single CPU VMs on the same host. Eg. All hosts are 1vCPU or all 2vCPU etc.

[RobinVm1]

Thanks Callifo,

i think i'll give that a try.

Robin

[Jonathan33]

I am currently dealing with CPU contention on Windows XP work stations on a 3.5 box and have noticed a couple things. Machines running Visual Studio or SQL behave better and have less contention at 2 CPU. The lower CPU count lowered the CPU ready numbers in my case. I have not attempted to use VMware View yet, but it would appear that is a product that could be in line to help with such virtual desktop issues.

[RobinVm1]

Well, just for Feedback:

I configured all my VMs to have 2 vCPUs.

Now i experience rarely "Average CPU Ready" above 3%.

I think i'll keep that in mind for futher configurations.

Thanks to all who took the time, to help me about this issue.

[frank_wegner]

I have seen a similar case where only half the hyperthreads were used on an HP blade during high load. We have found a solution in http://kb.vmware.com/kb/1018206: Check the BIOS of your ESX host. The HP Power Profile Mode should be „Maximum Performance“.

In our case it was set to "Balanced Power and Performance". Switching tio "High Performance" did wonders.