Disk Space time remaining 0 days... but what VMs.

[coafark]

Hello all.

  I installed the vAPP for VCOPS about 2 weeks ago and have been going thru it trying to learn it.

I've been looking at alerts and trying to get the health up.   Most things are clickable, but for this I can't figure out what/where to go.

If I highlight something on the left (my vCenter for me), then go to Planning on the right, then Views.  

I find a lot of RED things.  For instance if I pick "Virtual Machine Capacity", then down below I'll see next to "Disk Space" that I have Time Remaining "0 days".

Also that 27 of my VMs are over.   I can't figure out how to find just what VMs those are.

I've attached a screen shot showing the page.

vCops   5.7.1  build 1166139

Thanks a lot.

[mark_j]

If I was a betting man, I would say that you're: 1. Actively thin provisioning disks and 2. Have the the configuration policy configured in tab 3a to Use the Allocation Model for Disk Space.

To fix this, disable the Disk Space / Allocation model (and use demand model for this) for the config policy applied to these resource and consider adjusting the overallocation #s for disk space on 3c.

Remember, the capacity planning functions will only be as accurate as you configure them to be for your environment.

[coafark]

Thanks for writing back Mark.  You were 1/2 right, 1/2 wrong.  I do have the defaults "by demand" and "by Allocation" enabled (see Screen shot).   I also only do thick, so no thin provisioning.  They are FC LUNs, does this matter?

With your comment, I was able to dig into this idea and it appears that I want both set.  I found this site, that explains it nicely.

http://imallvirtual.com/?p=761

We use vendor white paper specs, so we have a lotta waste (vCPU, memory) most times.  So I don't want to see alerts on this all the time.  From what I read, Demand will hammer me on that, right?

-Christopher.

[mark_j]

The article you have there is valid for some situations, but I was specifically focusing on your dis space issue and if thin-disks were used.

Are you using physical capacity or usable capacity in 3a? If usable capacity, let's also have a look at your buffers for disk space on 3b.

Do you have any RDMs? In the screenshot you sent, what was in focus on the left side, World / vCenter / Datacenter/ Cluster / etc?

Basically, something is dropping your VM capacity down very low for Disk Space (1.5 VMs). We need to identify the cause.

[coafark]

I'm using Physical Capacity.

  Even tho physical, I'll list buffers on 3b

      Use high availablilty (HA).... is checked.

      Nothing else.

  Use last known capacity (radio button checked)

No RDMs in this vCenter.   My other vCenter that I had pulled in then removed had some large ones.   I checked to make sure that the datastore that held them (its from version 4.1 vCenter) isn't in this vCenter (5.0) and its not.

I was highlighting my "vCenter"

[mark_j]

Have you confirmed that vCOps has proper permissions to vCenter Datastores? I'm wondering if it cannot properly calculate the capacity, causing the 1.5 VM capacity.

Let's do this.. export a "Datastore Inventory" report so we can see how much capacity it "thinks" you have.

[coafark]

Found it.

Planning / Views.

Found it now.  Attaching.

[mark_j]

Go to the View for Datastore Inventory and hit "export" while in focus of the WORLD or vCenter top-level.

[coafark]

ofc I found it after I posted I couldn't find it.  I edited my last post with the upload.

[mark_j]

You've got some weirdness going on with how your disk space is reporting.

You've got disk space that is provisioned, but it's not being reflected in # VMDKs or VMDK Disk Space used.

I think there is a problem with ether how your hosts are reporting (missing stats reporting) or your permissions to the vCenter objects.

Check to ensure every single host is reflecting performance metrics and not 0 CPU/ 0 Memory. Also, do yourself a favor and generate a new service account in vCenter, give it Global: Health and Storage View : Views at the root level with prop, and config vCOps to use that as the collection account. This will ensure the stats CAN be collected by vCOps to generate appropriate planning #s.

[coafark]

I'm logging in to VCOPs with the administrator account that runs vCenter today.

Wall of text time 🙂

Does it dynamically re adjust these reports (planning / views/ etc) or do I have to do something to get it to use that account?

To reduce what we're looking at, I started looking around on the "left" side and it appears that ANY cluster object that has a host in it, has the disk space issues.   So I'll highlight one cluster "DataCenter1" that only has ONE HOST in it.

Capacity Risk Details

Disk space  5.9TB  usable cap 5.9TB, cap remain 0% over by 28 VMs

I only have ONE VM on this host.   44 LUNs are viewable on it.

I opened up vCenter to look at the VM.  It has 3 drives (all local drives on the host).

DOCPVS1a

DOCPVS1a_1

DOCCtxPV2-00001.vmdk

Not sure where the performance metrics are, but here is what I can find.

While highlighting the host,  Operations/ all metrics (open health tree).

CPU Utilization for resouces/CPU Active (%)(1min. average)  that's fluctuating up and down.

Two thoughts

1)  We have a parent agency that backs up our VMs with Netback up.  It does snap shots then deletes them.  Can this "usage" be confusing vCOPs??

2)  We are transitioning from one vCenter (4.1) to this vCenter (5.0).  So we have LUNs that are viewable in both vCenters.  Can LUNs that are viewable in this vCenter (vCops sees) BUT the VMs are running on the other vCenter be effecting this?  vCops would see the usage, but not have the vmdks in its inventory???

[mark_j · Accepted solution]

If you're using the vCenter admin account for the vCOps collector, perms probably aren't an issue in this case. vCenter creds are only used to collect data from vCenter, and for user access it acts as a pass-through for access to the Advanced UI. Either way, I don't think it's an immediate issue since you're using your admin account.

1. Snapshot space on a VM will just count towards disk space usage. After a certain age it is even considered waste (configurable, 180day default).

2. That would absolutely be impacting the accuracy of what vCOps sees. vCOps needs to see all VMs on a datastore, and giving it only partial access even if across vCenters can cause serious blind-spot situations with these planning #s. I recommend adding the 2nd vCenter to vCOps for monitoring in order to help clean this up, however you may find the datastores are the same signature, but with 'possibly' differing vCOps resource name - see what it does in your case. When you migrate VMs to the new vCenter, it'll gen new unique IDs for the resources and you'll be losing your historical data for those VMs. The VMs in the new vCenter will be collected as new resorces and collect metrics under that new resourceid... the old resource and associated metrics will be left to age out with the rest of your data.

How long until this migration is finished? Until you get your datastores cleaned up and VMs on them fully visible to vCOps, I fully expect the #s will be a bit off.

[coafark]

Wow, so I threw myself under the bus there and didn't know it.

Glad to see that its nothing wrong on my side, just incorrect sharing of LUNs that is doing this.

I can't add another vCenter, I only have 1 license and our upgrade wont happen soon enough for a trial license.   Our existing environment (4.1x) has 45 hosts, 6 of which are 3.5esx.

We have a vendor coming in to help upgrade our remote sites.  From 3.5 or 4.1 to 5.5 (and hardware).

To make sure I don't drag any garbage thru, I'm going to install a new vCenter (5.5) and import each host as it is upgraded to 5.5.  This way, I don't carry thru remnants of (3.5-->4.0-->4.1-->5.0-->5.5).

I'm not worried about the historical data that I've collected with this vCOPs.  Its only about 2 weeks old and is the vAPP.   Unless at 5.5 things have changed a lot,  VMware recommended separate LUNS for the Analytics VM and the UI VM.  The 5.0 vAPP puts them all on the same storage.  So I was going to build them out (at 5.5) rather then using a vAPP.

I really appreciate all your help with this.

[mark_j]

No problem, I'm glad I could assist.

BTW, there is no harm is keeping the vApp on the same LUN. The only reason you probably heard otherwise is for administrative organization or distributing I/O load. At long as you have the IOPS on the back-end and an adequate fibre or network pipe through to the hosts, it doesn't matter how you arrange the datastores. If you have a VNX w/FAST or VMAX.. you probably won't care if it's on the same LUN. If you've got a SMB or entry-level SAN with spotty pools of spindles, then you may want to split up your VMDKs to LUNs to distribute over different spindles. So if you have the IOPS and Diskspace.. no harm throwing them on big datastores.

One suggestion that I would share on placement/sizing, is to add VMDKs of reasonable size (I usually do chunks of 250 or 500 depending on the size of overall deployment). I say this because I prefer to keep the VMDKs managable so that I can svmotion them around within a reasonable amount of time and not need to wait overnight for a 1-2TB migration.

[coafark]

I think it was for the IOPs.   A parent agency provides the SAN for us, but its a Hitachi FC, so it should be fast enough for a combined one.

I normally don't chunk out large areas for multiple VMs. I keep it to 1 LUN per VM so when I move storage, its only 1 VM that gets effected.   Only multiple LUNs when its something like a high utilized Database, so I don't have IO contention.