ESXi v4 HP ML370G6 Performance Question - Nehalem has HT, Run 16 cores or 8 cores?

[GregBradley]

I'm about to go live with my latest system for a customer. This G6 is just amazing in performance even compared to the G5 that I use for my own business. Nothing was clear on the pros and cons of enabling Hyperthreading on the Xeon E5530. With 2 QuadCore CPUs, HT enabled gives me 16 cores. The free version of ESXi gives me a max 4 cores SMP for any machine I create.

I'm following what has worked well for other customers on the G5 and just doubling the CPUs on most virtuals. This leaves me wondering if I gain anything by NOT using HT. 24Gb RAM, P410i/512 BBWC, 5x146Gb 10K DP ENT drives:

• 1 core: Win2003x32 for AD

• 2 core: Win2003x32 for AD backup and print server (16 printers)

• 4 core: Win2003x32 for a poorly behaved 16bit Accounting Software that now won't require shutting off other stuff to reboot the ancient crap!

• 4 core: Win2003x64 for file server - on its own Gb Enet thanks to the 4 port that comes with the G6 - almost no traffic to other servers

• 4 core: Win2003x32 & SQL Workgroup

• 2 core: Win2003x32 & Exchange 2003 for half the company - due to 16Gb limit on store

• 2 core: Win2003x32 & Exchange 2003 for half the company - due to 16Gb limit on store

• 2 core: ClarkConnect (ClearOS) (Redhat 5) firewall, AV, Antispam, mail server, ftp server.

It seems to work well in my testing and the two Exchange servers work much better than the dedicated ML310G3 units that are now turned off. There are a dozen heavy users and a dozen more. A bunch of remote users that mainly use the groupware on the Clarkconnect box.

Any input welcome before I go live this weekend.

[DSTAVERT]

HT isn't quite the same as a core. You will probably end up with scheduling problems for the 4 vCPU machines since VM will need to wait until all 4 vCPU's are available before it gets any. I would be a little more selective in assigning multiple processors. Can the accounting system survive with a single CPU?

Someone may have a different experience. There are lots of 4 CPU vs 1 or 2 CPU posts in the forums.

http://kb.vmware.com/kb/2032

http://kb.vmware.com/kb/1005362

[J1mbo]

Agree with the above, I would suggest more like,

1 vCPU: Win2003x32 for AD

1 vCPU: Win2003x32 for AD backup and print server (16 printers)

1 vCPU: Win2003x32 for a poorly behaved 16bit Accounting Software that now won't require shutting off other stuff to reboot the ancient crap!

1 vCPU: Win2003x64 for file server - on its own Gb Enet thanks to the 4 port that comes with the G6 (??) - almost no traffic to other servers

2 vCPU: Win2003x32 & SQL Workgroup

2 vCPU: Win2003x32 & Exchange 2003 for half the company - due to 16Gb limit on store (??)

1 vCPU: ClarkConnect (ClearOS) (Redhat 5) firewall, AV, Antispam, mail server, ftp server

Once in MONITOR these VMs to see which hosts NEED additional CPU resource. Bear in mind the processing cores of these new chips are super-quick, and as said above by far the biggest issue is scheduling.

Say you have a 4 vCPU VM. ESX will schedule CPU resource, for up to 30ms, of four cores (HALF your capacity). That VM might be slightly busy, say with two threads, leaving two cores running the Windows IDLE thread instead of doing helpful work for another VM, until ESX pre-empts it's use of the processor.

Also multiple vCPUs will only provide a benefit to applications with proper multithreaded workloads. 16-bit software very probably isn't. SQL & exchange might benefit depending on workload (not SQL Server Express though).

Finally re hyper-threading, yes this can offer benefits as ESX is specifically aware of the technology, wikipedia article describes it well:

When execution resources would not be used by the current task in a processor without hyper-threading, and especially when the processor is stalled, a hyper-threading equipped processor can use those execution resources to execute another scheduled task. (The processor may stall due to a cache miss, branch misprediction, or data dependency.)

I emboldened that network point, because this is likely not the optimum configuration. Assign multiple NICs to vSwitches as required for the overall bandwidth, and look at allocation algorithms if required (port, MAC hash, IP hash) - this will ensure a fault tolerant configuration to all hosts, rather than an arbitary split of resources.

Secondly what is the 16GB limit refered to?

Hope that helps.

Please award points to any useful answer.

[admin]

Each HT twin on Nehalem behaves like 10% to 15% of an additional physical core. For allocation purposes, you should treat a dual quad core HT Nehalem system as if it has ~9 physical cores.

[GregBradley]

Much helpful info so far - thanks to all.

The 16Gb limit refers to the mailstore max size in Exchange Server 2003. Enterprise removes that limit as does Exchange 2007. They grew over 10 years from 1Gb to 4Gb, but then in another year grew to 18Gb, forcing the addition of another server since they wanted it fixed instantly. We just split the users over two physical server that were BOTH maxed out within a year, now spread onto the hosts on the ESX unit. Moving mailboxes saves enough space that they are happy for now. If they won't fix the problem which is caused by excessive cc of emails with large attachments, I will go to Exchange 2007. I am replacing the physical server with the virtual units using the server software already licensed.

When you email someone and include a copy of the document instead of emailing a link it makes duplicate copies of the attachment. When you cc 8-10 people on every update, it gets ugly fast. I could probably find examples of a huge excel sheet where there are now 300 copies of the attachment in the inboxes, which the users don't dare remove, ever! The owner just asked me to set something up where he is automatically copied on every email in and out of the business. His mailbox is already 8Gb. Obviously, I won't even try to do that.

I am running it now with HT enabled so ESXi shows 16 CPUs available. I assumed when I started that I would have 8. SQL only licensed for 2 CPU so that might influence the decision to have 8 CPU or 16.

[J1mbo]

When I looked into it, the SQL licensing become per vCPU in the virtual world...

Please award points to any useful answer.

[DSTAVERT]

Exchange 2003 can have a 75GB Store.

http://technet.microsoft.com/en-us/library/aa998066%28EXCHG.65%29.aspx

[DSTAVERT]

Carefully read jmattson's post and don't look at that total as 16. That might be great for VDI but this is a server environment and a 10th of a processor isn't a processor.

Also you can change how processors are presented to a VM and used by ESXi. From the VI client you can present the VM with a single processor with 2 cores.

http://www.yellow-bricks.com/2009/06/04/per-processor-licenses-for-your-application/

[DSTAVERT]

I haven't worked with Exchange for a year or so but it doesn't store attachments or email as single items. From the MS KB http://support.microsoft.com/kb/143376

Microsoft Exchange performs single instance storage in the database on a Microsoft Exchange Server, so the 16GB limit on the private database is really a much larger logical storage. Single instance storage means that if 1000 users on a Microsoft Exchange Server get the same message, the message is only stored once in the database and each of the users gets a pointer to the message. The same single instance storage also holds for attachments. If a user opens their copy of a message and modifies it but they don't modify the attachment, Microsoft Exchange doesn't resave the attachments, just the new message body. 

[GregBradley]

I totally missed when Exchange 2003 got SP2 that it could be configured to have a larger store. I do recall that the limit of 16Gb could be exceeded and it would slow down gracefully instead of just going boom like Exchange 2000 did. With hindsight and this info I could have expanded the store on the old box but I had the extra box from a defunct customer. I'm glad I didn't tell my customer since the mail issue is what drove the ESXi upgrade that was needed for other reasons. Email attachments are not stored individually but 300 attachments of the same file still take up 300 times the space, right? It seems it doesn't according to the info above.

Sounds like I should consider the box to be 8 CPU or so, like the G5 units (2x Xeon 5440) . Does this mean that I should turn off HT in the BIOS? Does this give me 8 "faster" CPUs rather than 16 "slower" ones?

It seems like the x64 File Server might work quite well with a buch of RAM allocated and one CPU. In the current box it has some other functions. In the ESX change, I will have it doing ONLY file server functions.

The info on ESXi cores and processors and sockets may benefit my next install who actually uses SQL instead of just getting a custom application going. Thanks!

[admin · Accepted solution]

Sounds like I should consider the box to be 8 CPU or so, like the G5 units (2x Xeon 5440) . Does this mean that I should turn off HT in the BIOS? Does this give me 8 "faster" CPUs rather than 16 "slower" ones?

There is generally no advantage to disabling HT on Nehalem (in stark contrast to earlier HT implementations).

[DSTAVERT]

No need to turn off HT just ignore the claim that you have more cores. Configure VM's with 1 CPU unless in testing you find it needs and can use more. Don't starve the VM for RAM or CPU but know that they need the resources before you give it to them.

Exchange doesn't store multiple copies of email OR attachments. Emails and attachments are stored in the database and each user only gets a reference to the instance. If an email is forwarded to another user on the system the email is changed but it only contains the pointer to the original attachment. If the forward goes to 10 users there is only 1 copy of the changed email not 10.