We have decided to move our Oracle Forms servers to VMs. We built a 2 vCPU with 4GB of Ram Windows 2003 SP2 VM on a new ESX 3.5 host using its own LUN on a EVA8100. Our DBAs started to do some forms compiling on the VM and noticed that it was taking around 4 times longer on the VM than it does on physical servers. They wanted me to add more CPUs so I added 2 bringing the server up to 4 total. I watched the performance as they reran the forms compiling and noticed that the actual process doing the compiling was only using about 25% of the CPU (my thinking is that the process does not take advantage of multiple cores). I put the server back down to 2 vCPUs and after rerunning the compile it was using about 50% CPU (adding to my suspicion that it does not take advantage of multiple cores).
Since it was a new Oracle install we thought maybe there is something diffrent between the physical server and the VM, so I did a test with a script that gathers all the email addresses in our Active Directory and puts them in a file. I added a function to the script to put a time stamp at the top and bottom of the file so I could compare between the physical and virtual server. The physical server finished the script in around 6.5 min but the virtual server took around 16-17 min to finish the AD script.
Since we were about to upgrade to ESX 4 we decided not to spend much more time figuring this out. Our upgrade to ESX 4 is now complete and I have upgraded the VMware Tools and Hardware of the VM to the latest versions. It still takes around 16-17min to run the AD script.
I then built a brand new VM with 1 vCPU and 1GB of Ram in the same LUN as the origional server and installed Windows 2003 SP2 onto it and installed the VMware tools. After that I ran the AD script 5 times and it finished on average at just under 5 minutes. I then installed all available WIndows Updates and ran the AD script again with no significant time increase.
Next I shutdown the VM and added a vCPU. After finding the new hardware I ran the AD script 5 more times with an average around 17min. I watched the server with Process Explorer and the Hardware Interupts are running around 20% the entire time the script is running. I used KernView to see what was going on in the Kernal of Windows and I get processor, ntkrnlpa, hal, tcpip, win32k, e1000325, ndis, afd, fltmgr, and ntfs as the top 10 items.
Using Process Explorer on the Physical Server shows the Hardware Interupts around 2% at the highest.
I have gone so far as to make this the only VM running on an HP BL465 G5 with 2 Quad Core AMD 2.7GHz processors and 32GB of ram and setting up CPU and Memory reservations to what the VM has but it does not perform any better. Does anyone have any idea how to get better performance out of a VM with multiple vCPUs? Or why the Hardware Interrupts are so high?
