gaps in real-time performance data metrics from vcenter 4.1.0 server

[bergerpjr77]

Per the enclosed picture of the real-time performance metrics -- we seem to have repeating gaps in performance metrics data. this "only" happens on the real-time view -- the roll-up (daily; weekly; monthly; yearly) charts/views are just fine with no gaps.

i did some research and ruled out/tested the following:

• esx hosts and vcenter are the same date/time/ntp settings.
• connected directly to the esx host and looked at real-time performance metrics. same gaps as in the vcenter view.
• gaps appear for all vm's on all hosts.
• checked out the vcenter "vpxd####.log" files -- no problems.
• checked name resolution between vm; vc and esx hosts. no problems.
• checked out vmware kb articles: kb1883; kb1925; kb9960761 and kb1003878.
• back-end/remote sql server is not overloaded.
• c:\programdata\vmware\virtual center server\logs = checked. no problems.
• checked sql to make sure sql roll-up jobs are all running fine.
• vcenter database has a maintenance job that defragmentsindexes it nightly.
• all stored procedures/jobs are present. database administrators says sql server looks fine to them.

i've been all around the internet searching for root cause. many of the articles talk about older esx/vcenter 2.x issues -- however we're running vCenter server v4.1.0 (build 491557) on 2008r2 and sql 2008 on 2008r2.esx is v4.1.0.381591

any tips/tricks to figure out/stop these performance metrics gaps would be great. thanks!

[EdWilts]

Do your hosts also show "Stats insertion failed for entity <hostname> due to ODBC error."?  You'll find this on the the Tasks & events tab under Events.

If so, it's due to a bug in vCenter and is supposedly fixed in 5.0 Update 1.  We see this regularly even in 5.0 but haven't been able to go to Update 1 yet.

There are things you can do to make it better - faster db hardware, reduce the db table sizes, etc. - but we haven't been able to make the problem to away completely and we're really hoping that U1 does fix it.

[bergerpjr77]

Ed -- thanks for the head's up -- however there are zero entries on any of the hosts for "stats insertion failures" or anything relating to failures relating to stats/ODBC/sql/performance, etc.

The gaps in the real-time performance data are usually short 3-4minutes -- so it's not hard for a human to continue to plot the missing data and get the "general idea" about performance, but the fact that its recurring/frequent is discerning.

As for the back-end SQL hardware -- it's uber-beefy. 60Gb RAM; AMD Opteron 6174 (2.19Ghz x4 (48 vCPUs).

As for disk -- the DBA's did it pretty well -- it is on the NetApp SAN and there are seperate drives for: OS; Logs; SystemDB; UserDB; Temp; Pagefile.

Network pipe is 1000Gbps.

thanks...

Message was edited by: bergerpjr77 - fixed the missing decimal point in the CPU speed. 219Ghz would be nice though.

[bergerpjr77]

Looks like this article might help too:

http://kb.vmware.com/selfservice/search.do?cmd=displayKC&docType=kc&externalId=1030819