HTTP Status 500 - Internal Server Error

[2eto]

Hello all,

I recently had an incident in my server room where the AC was switched off to all the racks and much of the kit had to be shut down to prevent damage. Since that incident I am no longer able to log into the VSphere managment console, and none of the backups of my virtual machines are running successfully. Veeam says it cannot see any VM severs. However, the the host servers are up and running, I can log into them all via RDP. I can log into the ESX Host servers too.

I can log into Vcenter management console, and in the summary I have green check marks beside overall health status. I'm running Version 7.0.3.01400

I've read that the above error is related to expired certificates. I've read a few articles on how to check whether certificates are expired. I've logged into the Vcenter VM as root, and none of the commands I see posted in the articles that deal with this matter or work, such as;

for store in $(/usr/lib/vmware-vmafd/bin/vecs-cli store list | grep -v TRUSTED_ROOT_CRLS); do echo "[*] Store :" $store; /usr/lib/vmware-vmafd/bin/vecs-cli entry list --store $store --text | grep -ie "Alias" -ie "Not After";done;

Or the command to launch the certificate manager.

Any assistance would be greatly appreciated.

[Ardaneh · Accepted solution]

Hi

HTTP status 500 error has lots of reasons, so please check the items below:

- Check the certificate status by using your browser

- Check for vCenter storage free space by CLI (sometimes VAMI gives inaccurate information!)

- Check all services' status by using this command: service-control --status

- Check the VPXD log file for more information

[2eto]

Hi Ardaneh,

Thanks for the reply, and providing a concise list of items to check.

• Certificate is expired, checked via the browser.
• Checked the services status. Screen shot of whats running and not running in the attached.
• Checked free space, another screen shot.
• How do I check the VPXD logs on the CLI?

Cheers!

[battybishop · Accepted solution]

The first thing you'll need to do is sort out your certificates

check out this article

HTTP Status 500 – Internal Server Error vCenter (vmkfix.blogspot.com)

[2eto]

I reset all certificates and now I have access to Vsphere again.

Thanks for the responses and the useful feedback.

[battybishop]

Excellent, pleased to hear your back up and running 😁

[mairabano]

I'm sorry to hear about the issues you're facing with your VMware vSphere environment. It does sound like certificate-related problems could be causing your login issues and backup failures. Let's try to troubleshoot this:

1. Check Certificate Expiry: It's essential to ensure that the SSL certificates in your vCenter Server are not expired. Since the commands you've mentioned don't seem to be working, you might want to try using the VMware Certificate Manager utility (certificate-manager). This utility allows you to review and update certificates.

   To access the Certificate Manager:

   • Log in to your vCenter Server Appliance using SSH.
   • Run the command: certificate-manager

   From there, you can navigate the menu to view and manage certificates.

2. Verify vCenter Services: Make sure that all vCenter services are running correctly. If any service has failed to start, this could be contributing to your login issues.

3. Check vCenter Server Logs: Review the logs in /var/log/vmware/vpxd/ on your vCenter Server Appliance. Look for any error messages or warnings that might provide more information about the issue.

4. Veeam Configuration: Regarding Veeam, ensure that the Veeam server can reach the vCenter Server and ESXi hosts. Check the Veeam configuration and make sure the credentials are correct. Also, verify that the Veeam services are running without issues.

5. Backup and Snapshot Removal: In case your backup issues are related to snapshots, you may need to remove or consolidate snapshots for your virtual machines if they were left in an inconsistent state after the incident. Be cautious when doing this, as it can affect VM performance. Projectors Focus

6. VMware Support: If you're unable to resolve the issue through the steps above, consider contacting VMware support for further assistance. They can provide more advanced troubleshooting and help you resolve certificate-related problems.

Remember to perform any actions carefully, especially when dealing with certificates and snapshots, as improper changes can cause further issues. It's advisable to have backups of your vCenter Server and critical VMs before making any significant changes.