VMware Cloud Community
andvm
Hot Shot
Hot Shot

VM Memory Usage - Obtain correct metric from vSphere

Hi,

VM has 24GB of Memory.

From vSphere I see very low memory usage:

andvm_1-1662474005302.png

 

From vROPS I see 100% memory usage

andvm_0-1662473976509.png

I know that the VM is struggling with memory but why I am not seeing this from VM view in vSphere? (where should I be able to see if/which chart) ?

 

Thanks

 

 

0 Kudos
3 Replies
bryanvaneeden
Hot Shot
Hot Shot

You can see the VM metrics under the Monitor/Performance tab. Are we looking at the same VM? 

 

Also have a look at: https://kb.vmware.com/s/article/55675. Not sure if you are hitting this, but it confused the hell out of me.

Visit my blog at https://vcloudvision.com!
0 Kudos
Tibmeister
Expert
Expert

Is this a Windows or Linux VM?

The short answer is, the Guest OS will see different metrics than the hypervisor because of the way a hypervisor works on abstraction of the hardware.

One of the main things that trips folks up, especially when dealing with Windows, is the way Windows looks at RAM usage vice what is reality, and how it manages RAM fundamentally.  Unless you really dig, you will not see the memory breakdown of Zero Pages, Free Pages, and Cache Pages.  In Windows, these all are added up to determine the "used RAM" for the OS.  In reality, Zero Pages are pages in RAM that are not used, have no active data written to them, but Windows doesn't want to actually release them because it's easier to just ignore unless the OS is under true memory pressure.  ESXi will not allocate Zero Pages in the hypervisor, and it will also not count this as the VM Consumed RAM because there is no active data being written to that memory address, so that will be made available to another VM on the system unless you have reservation in place.  There's many articles on this, it's not a new phenomenon, and systems running applications like Exchange or SQL that will "consume" all available memory and write zero's to what is not currently needed, Windows will see this as consumed by the process where ESXi will see these as Zero Pages and not allocate it.

If you are simply looking at Windows Task Manager, WMI, or SNMP, and see that RAM utilization is "high" and thinking there's memory pressure, than you are not looking at the full picture.  You have to be careful which metric you are looking at in vROPs, because looking at the in-guest metrics will give you the same false picture as using Task Manager or SNMP.

So, is there a lot of paging activity happening in the Guest OS?  Look at the vROPs stats from the VM Guest Metrics, not the Windows Guest Metrics.  This is very much different as the VM Guest Metrics get's data from VM Tools and distinguishes between the Zero Pages, and the latter get's the data from WMI, which sums the Zero Pages into it's reporting.  vROPs does have a Zero Page metric, and if you wanted to do the math by hand, I bet you will find, within a margin of error, that things match up and make sense.

So after all that jabber, what your chart is telling me, just shooting from the hip, is that you have an application that is consuming all available RAM in the OS just because it can, and is writing zero's to the RAM just so that the OS sees it's allocated to a process.  ESXi is not seeing that the OS is really using all this because it's ignoring these Zero pages.  So really, your OS is not even using 1GB of RAM, much less the 24GB assigned to it.  I would look at the application to make sure there's no memory leak and that it can be configure to use a fixed set of RAM.  Look in vROPs for the VM Guest Metrics and VM memory used charts to get a better picture, and look for the Zero Page charts.  I often used them to go to the app owner and let them know they are wasting resources because the Zero Page is telling me how much RAM is over-allocated to the VM just because.  Often, it's because "a vendor said so", which unless the vendor is footing the bill for your infrastructure, they don't get to waste your resources.

andvm
Hot Shot
Hot Shot

Very interesting and detailed feedback.

VM OS is Windows, ram had been increased to 32GB and seeing that Chart Memory|Guest Usage (KB) is showing all of them in use.

However looking at Guest|Used Memory (KB) chart only approx 3GB are used.

Is this a reflection of what you said i.e. first graph inclusive of Zero Pages and second one exclusive?

Not sure which chart I have to look for Zero Pages one as only seem to see Memory|Non Zero Active (KB) which looks to be less than 1GB

andvm_1-1663135075951.png

 
andvm_5-1663135220079.png

 

Very tricky if so to identify and also to explain to virtual machine owner.

Also these 2x charts are very similarly named which does not make it easy to understand their differences.

0 Kudos