New vm Server much slower then physical server

[miklaw2]

We have a server that runs one app.  It is a processor application that reaches out to a SQL database to see if there is a record for it to work on and then if so visuall shows the record number that it has recieved on a the window on the server process window and connects to a SQL database on a different server and runs some stuff and when complete updates the process window that its done and goes to the next record or sleeps for 5 seconds and checks again.

On the physical server this process takes about 1 - 3 seconds per record.

Did a P2V and on the VM the process took 8 - 11 seconds per record 

hmmm

So we rolled back as this was putting us behind during the day time hours...

and built a new vm with windows 2003 and installed the app on it.

still took 8 - 11 seconds

moved it from our new EMC Celera (NFS) Datastore to our NetApp 3050 (NFS) Datastore

no change

moved it to a different host

no change still slower then physical

hmmm

it had a E1000 nic so I switched it out for a VMXNET3 nic

still takes 8 - 11 seconds.

Hosts doesn't appear to be taxed in anyway... other VM's appear to be behaving...

More info if you have read this far...

Virtual Center -  4.1 build 258901

Host - VSPHERE Enterprise 4.1.0,320092 on all hosts

Vm - VM version 7 , VM Tools 8.3.2 Build -299420   OS: Microsoft Windows 2003 SP2

not sure what else would help you give me a kick start in the right direction...

Thanks in advance for any help/insight you can provide.

[mittim12]

Have your reviewed esxtop while the process is being run to see if there is any contention from the VM side?  Maybe high utilization of process, high %ready times, or maybe saturation of the disk.    

[lowteck]

are the cpu and memory resources you allocated simliar?

if it is in a resource pool, what are the settings?

low

[miklaw2]

No I havent...  to be honest my experience with ESXTOP started yesterday... I had not used it before and was reading through a performance troubleshooting doc that I had downloaded.

But I will certainly give that a try... I am not always sure what I am looking at with ESXTOP... but hopefully something will jump out.  thanks

[miklaw2]

Yes the setting are same or better

everything currently is using the same pool... all default

[lowteck]

try bumping up your cpu memory and resource settings to high.

you could also try reserving both resources as well.

low

[mittim12]

here is a good document that list the counters and definitions.   http://communities.vmware.com/docs/DOC-5240

I like to look at %RDY, DAVG/cmd,QUED

[Exwork]

Look at the CPU performance charts in the VI client. Does the guest VM have only 1 vCPU? If so, are you maxing it out?

From inside the VM, open Task Manager, and watch the CPU usage when you run the query.

[miklaw2]

Lots of things to try... neither the physical or the virtual use hardly any cpu or memory... and I did look at that through the gui and the cpu was almost flat... and the memory was less then 1/2 of the 2gb allocated...

I have somethings to look at... and learn...

Thanks! 

[Exwork]

How many vCPUs did you give the guest?

[cody_bunch]

On both the physical & virtual fire up perfmon or sar -w and look for the number of context switches per second. They are the silent killer. More info here: http://professionalvmware.com/2010/11/context-switching-some-resources/

Second, use esxtop on the host and make sure you're under the thresholds here:

http://www.yellow-bricks.com/esxtop/

Finally, you will want to use this doc as a guide for troubleshooting:

http://communities.vmware.com/docs/DOC-14905

That should at least get you headed the right direction.

-Cody

http://professionalvmware.com