Vcenter 6.0u3 inaccesible / Certificates Issue

[i4pcoil]

So, yesterday I can over my Vcenter to do some work and discovered I cannot login using the HTML5 console with the error

[400] An error occurred while sending an authentication request to the vCenter Single Sign-On server - An error occurred when processing the metadata during vCenter Single Sign-On setup - Failed to connect to VMware Lookup Service https://vcenter/lookupservice/sdk - SSL certificate verification failed..

SSHing the VCSA appliance and trying to start all services yields:

2021-05-01T09:11:31.413Z RC = 1
Stdout = Starting VMware Inventory Service...
Waiting for VMware Inventory Service....................................................................................
WARNING: VMware Inventory Service may have failed to start.

Stderr =
2021-05-01T09:11:31.414Z {
"resolution": null,
"detail": [
{
"args": [
"Command: ['/sbin/service', u'vmware-invsvc', 'start']\nStderr: "
],
"id": "install.ciscommon.command.errinvoke",
"localized": "An error occurred while invoking external command : 'Command: ['/sbin/service', u'vmware-invsvc', 'start']\nStderr: '",
"translatable": "An error occurred while invoking external command : '%(0)s'"
}
],
"componentKey": null,
"problemId": null
}
ERROR:root:Unable to start service vmware-invsvc, Exception: {
"resolution": null,
"detail": [
{
"args": [
"vmware-invsvc"
],
"id": "install.ciscommon.service.failstart",
"localized": "An error occurred while starting service 'vmware-invsvc'",
"translatable": "An error occurred while starting service '%(0)s'"
}
],
"componentKey": null,
"problemId": null
}
Unable to start service vmware-invsvc, Exception: {
"resolution": null,
"detail": [
{
"args": [
"vmware-invsvc"
],
"id": "install.ciscommon.service.failstart",
"localized": "An error occurred while starting service 'vmware-invsvc'",
"translatable": "An error occurred while starting service '%(0)s'"
}
],
"componentKey": null,
"problemId": null

I am suspecting the SSL certificate expired as seen here in the invsvc log:

2021-05-01T09:11:28.804Z [WrapperListener_start_runner ERROR com.vmware.identity.interop.ldap.LinuxLdapClientLibrary opId=] certificate expired at [Fri Apr 23 06:07:17 UTC 2021]

but using the SSL certificates management for recreating a new certificate doesn't seem to work and I always end up with the invsvc not starting and the following error in the log:

[WrapperListener_start_runner ERROR com.vmware.identity.interop.ldap.LinuxLdapClientLibrary opId=] certificate expired at [Fri Apr 23 06:07:17 UTC 2021]

All these 4 commands show validity:

/usr/lib/vmware-vmafd/bin/vecs-cli entry list --store MACHINE_SSL_CERT --text | less
/usr/lib/vmware-vmafd/bin/vecs-cli entry list --store machine --text | less
/usr/lib/vmware-vmafd/bin/vecs-cli entry list --store vpxd --text | less
/usr/lib/vmware-vmafd/bin/vecs-cli entry list --store vsphere-webclient --text | less

Not Before: May  1 09:21:26 2021 GMT
Not After : May  1 09:21:26 2023 GMT

Any option using the /usr/lib/vmware-vmca/bin/certificate-manager  fails with rollback.. I managed to regenerate the certs and used Ctrl-C before the rollback

but this isn't helping much..

Following KB2112577 - #update_extension_certificate_on_vcenter_server_appliance

yields the error:

raise httplib.HTTPException("%d %s" % (resp.status, resp.reason))
httplib.HTTPException: 503 Service Unavailable (Failed to connect to endpoint: [N7Vmacore4Http16LocalServiceSpecE:0x7ff5dcf094b0] _serverNamespace = /sdk _isRedirect = false _port = 8085)

This is an Essential-Plus 6.0u3 cluster with 3 hosts. I'm not even sure what data is kept in the Vcenter database and if it won't be easier to setup Vcenter from scratch?

thanks,

    Justin

[Ajay1988]

What's the error  before it rollback ?  R u using option 8 in certificate-manager ?   Make sure to use proper inputs.
Is this with external  PSC or embedded? 

/usr/lib/vmware-vmafd/bin/vmafd-cli get-pnid --server-name localhost  (find PNID)- this is to be used 

[i4pcoil]

hi

/usr/lib/vmware-vmafd/bin/vmafd-cli get-pnid --server-name localhost
vcenter

Note : Use Ctrl-D to exit.
Option[1 to 8]: 6
Do you wish to generate all certificates using configuration file : Option[Y/N] ? : N

Please provide valid SSO and VC priviledged user credential to perform certificate operations.
Enter username [Administrator@vsphere.local]:
Enter password:
certool.cfg file exists, Do you wish to reconfigure : Option[Y/N] ? : Y

Press Enter key to skip optional parameters or use Previous value.

Enter proper value for 'Country' [Previous value : US] :

Enter proper value for 'Name' [Previous value : vcenter] :

Enter proper value for 'Organization' [Previous value : VMware] :

Enter proper value for 'OrgUnit' [Previous value : Machine] : Root

Enter proper value for 'State' [Previous value : California] :

Enter proper value for 'Locality' [Previous value : Palo Alto] :

Enter proper value for 'IPAddress' (Provide comma separated values for multiple IP addresses) [optional] : 172.17.1.100

Enter proper value for 'Email' [Previous value : email@acme.com] :

Enter proper value for 'Hostname' (Provide comma separated values for multiple Hostname entries) [Enter valid Fully Qualified Domain Name(FQDN), For Example : example.domain.com] : vcenter

Enter proper value for VMCA 'Name' :vcenter

You are going to regenerate Solution User Certificates using VMCA
Continue operation : Option[Y/N] ? : Y
Status : 40% Completed [Replace vpxd-extension Cert...]
2021-05-01T15:30:59.337Z Updating certificate for "com.vmware.vim.eam" extension

Status : 0% Completed [Operation failed, performing automatic rollback]
Rollback Status : 25% Completed [Rollback {0} Cert...]
2021-05-01T15:31:00.670Z Updating certificate for "com.vmware.vim.eam" extension

Error while reverting certificate for store : vpxd-extension
Rollback Status : 0% Completed [Rollback operation failed]

Error while performing rollback operation, please try Reset operation...

please see /var/log/vmware/vmcad/certificate-manager.log for more information.

[Ajay1988]

PNID is short name. That's means u didn't use fqdn during deployment .

The following one-liner can determine all expired certificates except sts:   
for i in $(/usr/lib/vmware-vmafd/bin/vecs-cli store list); do echo STORE $i; sudo /usr/lib/vmware-vmafd/bin/vecs-cli entry list --store $i --text | egrep "Alias|Not After"; done

To check for STS certificate : Use JXplorer to connect to vmdir with the following KB: https://kb.vmware.com/s/article/2146046
Expand IdentityManager > Tenants > vsphere.local
Select TenantCredential-1
Select Table Editor
Select non string data under the userCertificate
Select Details

If sts is fine. try  option 8 to reset all

[i4pcoil]

hi,

thanks for answering on weekend 🙂

I'm on vcsa appliance not windows...

used this https://kb.vmware.com/s/article/2146046 to get the certificates extracted and seems the websso and websphere-client certificates are expired:

Common Name: ssoserverSign Subject Alternative Names: vcenter Valid From: April 23, 2019 Valid To: April 22, 2021 Issuer: CA, vcenter Serial Number: ff1b3e97ffb2b5d9

Common Name: vsphere-webclient Subject Alternative Names: vcenter Organization Unit: mID-0d54d010-2ad0-4f18-8ec4-5790e31d1545 Country: US Valid From: April 23, 2019 Valid To: April 22, 2021 Issuer: CA, vcenter Serial Number: fd0ce666243139c4

thanks

Justin

[i4pcoil]

also,

for i in $(/usr/lib/vmware-vmafd/bin/vecs-cli store list); do echo STORE $i; sudo /usr/lib/vmware-vmafd/bin/vecs-cli entry list --store $i --text | egrep "Alias|Not After"; done
STORE MACHINE_SSL_CERT
Alias : __MACHINE_CERT
Not After : May 1 14:54:23 2023 GMT
STORE TRUSTED_ROOTS
Alias : 8a58237691549f44d3c18b2658bb4602cc7dd666
Not After : Apr 17 18:07:16 2029 GMT
Alias : 548c38856cba506ddaa27ffd9771cd78ce3bc3f1
Not After : Apr 20 17:43:47 2029 GMT
Alias : ac0bb7c4920da8697677262803553c9e51984e04
Not After : Apr 20 17:43:47 2029 GMT
Alias : 9ba26b9d3e351dbb283a20a999304d154cdec2b6
Not After : Apr 26 09:30:31 2031 GMT
Alias : 5fdcc8e3cc44822b454c3caed981e0371e371c8d
Not After : Apr 26 10:02:16 2031 GMT
Alias : ac3e8ed78bb664b23df5a263dbac40a049029613
Not After : Apr 17 06:19:56 2031 GMT
Alias : bf38cf0001cc9ae56c720e3ce43a5801ec3a1e37
Not After : Apr 26 12:01:54 2031 GMT
Alias : de002317cc672b3014f1b580b9db9d8960d64ce3
Not After : Apr 26 12:07:57 2031 GMT
Alias : c2a96eebec601f92bcc008e741d91ae6bd643bb0
Not After : Apr 26 14:07:02 2031 GMT
Alias : abb1ccf1c062a523c3f3c720cf740813e34eead7
Not After : Apr 26 14:31:15 2031 GMT
Alias : cd7f917513a541d31123fafa8a8bd7c1e43c4a1c
Not After : Apr 26 15:04:22 2031 GMT
Alias : b8e859038524a5a6502e6799b36840a9d0d9dbf1
Not After : Apr 26 15:25:40 2031 GMT
STORE TRUSTED_ROOT_CRLS
Alias : c63e389aceb3dab23a9f89db7b9753b9e36ef699
Alias : c13ddfca32c35ac9fb8b4694260a1c42d9345793
Alias : f653b6dd6b970f56a69fa2ca175e331c811c559b
Alias : 3e7b4a39c224b86fa9179bbe2ecb0d11421a474e
Alias : 49dbe9fa669966f5075c0da75853632fa030e9d2
Alias : d0ff91ce0ec976776d50e698dbb7cb5ff8fb422b
Alias : fe5956b0e5be77e844c46357fad75a7c83ca9f72
Alias : 47f2839946a46fc9c554e7cdd456a5e969cd7ecd
Alias : 65ed737444716190dd7cea5c2a910804d6ee1441
Alias : 6a96f6ab103889533c419a1205ed5124d01240c4
STORE machine
Alias : machine
Not After : May 1 14:55:15 2023 GMT
STORE vsphere-webclient
Alias : vsphere-webclient
Not After : May 1 14:55:16 2023 GMT
STORE vpxd
Alias : vpxd
Not After : May 1 14:55:16 2023 GMT
STORE vpxd-extension
Alias : vpxd-extension
Not After : May 1 14:55:17 2023 GMT
STORE SMS
Alias : sms_self_signed
Not After : Oct 4 11:26:11 2023 GMT
STORE BACKUP_STORE
Alias : bkp___MACHINE_CERT
Not After : May 1 14:54:23 2023 GMT
Alias : bkp_machine
Not After : May 1 14:55:15 2023 GMT
Alias : bkp_vsphere-webclient
Not After : May 1 14:55:16 2023 GMT
Alias : bkp_vpxd
Not After : May 1 14:55:16 2023 GMT
Alias : bkp_vpxd-extension
Not After : May 1 14:55:17 2023 GMT

[Ajay1988]

2021-05-01T15:30:59.337Z Updating certificate for "com.vmware.vim.eam" extension
Status : 0% Completed [Operation failed, performing automatic rollback]

All certs seems updated now. Run the below KB and check if that helps. 
https://kb.vmware.com/s/article/2112577

If sts is expired - https://docs.vmware.com/en/VMware-vSphere/6.0/com.vmware.vsphere.security.doc/GUID-497233EA-AEF9-464...

[i4pcoil]

https://kb.vmware.com/s/article/2112577 - been there 2 days ago and got error

python /usr/lib/vmware-vpx/scripts/updateExtensionCertInVC.py -e com.vmware.vim.eam -c /certificate/vpxd-extension.crt -k /certificate/vpxd-extension.key -s vcenter -u Administrator@vsphere.local
Password to connect to VC server for user="Administrator@vsphere.local":
2021-05-02T07:07:43.876Z Updating certificate for "com.vmware.vim.eam" extension
Traceback (most recent call last):
File "/usr/lib/vmware-vpx/scripts/updateExtensionCertInVC.py", line 174, in <module>
update_extension_cert_in_VC()
File "/usr/lib/vmware-vpx/scripts/updateExtensionCertInVC.py", line 139, in update_extension_cert_in_VC
sessionMgr = si.content.sessionManager
File "/usr/lib/vmware/site-packages/pyVmomi/VmomiSupport.py", line 537, in __call__
return self.f(*args, **kwargs)
File "/usr/lib/vmware/site-packages/pyVmomi/VmomiSupport.py", line 360, in _InvokeAccessor
return self._stub.InvokeAccessor(self, info)
File "/usr/lib/vmware/site-packages/pyVmomi/StubAdapterAccessorImpl.py", line 24, in InvokeAccessor
return self.InvokeMethod(mo, info, (prop,))
File "/usr/lib/vmware/site-packages/pyVmomi/SoapAdapter.py", line 1280, in InvokeMethod
raise httplib.HTTPException("%d %s" % (resp.status, resp.reason))
httplib.HTTPException: 503 Service Unavailable (Failed to connect to endpoint: [N7Vmacore4Http16LocalServiceSpecE:0x7fdcc4072040] _serverNamespace = /sdk _isRedirect = false _port = 8085)

Regarding STS regeneration seems the certs are OK:

/usr/java/jre-vmware/bin/keytool -keystore root-trust.jks -deststoretype JKS -list
Picked up JAVA_TOOL_OPTIONS: -Xms32M -Xmx128M
Enter keystore password:
Keystore type: JKS
Keystore provider: SUN

Your keystore contains 2 entries

root-ca, May 2, 2021, trustedCertEntry,
Certificate fingerprint (SHA1): xxxxxxxxxxxxxxxxxxxxx
newstssigning, May 2, 2021, PrivateKeyEntry,
Certificate fingerprint (SHA1): xxxxxxxxxxxxxxxxxxxxxx

The start all command returns:

2021-05-02T07:07:06.438Z Invoked command: ['/sbin/service', u'vmware-invsvc', 'start']
2021-05-02T07:07:06.438Z RC = 1
Stdout = Starting VMware Inventory Service...
Waiting for VMware Inventory Service.....................................................................................
WARNING: VMware Inventory Service may have failed to start.

Stderr =
2021-05-02T07:07:06.438Z {
"resolution": null,
"detail": [
{
"args": [
"Command: ['/sbin/service', u'vmware-invsvc', 'start']\nStderr: "
],
"id": "install.ciscommon.command.errinvoke",
"localized": "An error occurred while invoking external command : 'Command: ['/sbin/service', u'vmware-invsvc', 'start']\nStderr: '",
"translatable": "An error occurred while invoking external command : '%(0)s'"
}
],
"componentKey": null,
"problemId": null
}
ERROR:root:Unable to start service vmware-invsvc, Exception: {
"resolution": null,
"detail": [
{
"args": [
"vmware-invsvc"
],
"id": "install.ciscommon.service.failstart",
"localized": "An error occurred while starting service 'vmware-invsvc'",
"translatable": "An error occurred while starting service '%(0)s'"
}
],
"componentKey": null,
"problemId": null
}
Unable to start service vmware-invsvc, Exception: {
"resolution": null,
"detail": [
{
"args": [
"vmware-invsvc"
],
"id": "install.ciscommon.service.failstart",
"localized": "An error occurred while starting service 'vmware-invsvc'",
"translatable": "An error occurred while starting service '%(0)s'"
}
],
"componentKey": null,
"problemId": null
}

The invsvc.log still shows:

2021-05-02T10:06:27.993+03:00 [WrapperListener_start_runner INFO com.vmware.cis.lotus.LotusLocator opId=] Successfully refreshed machine account credentials
2021-05-02T10:06:28.001+03:00 [WrapperListener_start_runner ERROR com.vmware.identity.interop.ldap.LinuxLdapClientLibrary opId=] certificate expired at [Fri Apr 23 09:07:17 IDT 2021]
2021-05-02T10:06:28.002+03:00 [WrapperListener_start_runner WARN com.vmware.identity.interop.ldap.LdapErrorChecker opId=] Error received by LDAP client: com.vmware.identity.interop.ldap.LinuxLdapClientLibrary, error
code: -1
2021-05-02T10:06:28.002+03:00 [WrapperListener_start_runner ERROR com.vmware.cis.lotus.LdapUtils opId=] Failed to connect to LDAP; uri: ldaps://vcenter:636
2021-05-02T10:06:28.003+03:00 [WrapperListener_start_runner WARN com.vmware.cis.lotus.LdapConnectionFactory opId=] Failed to connect to LDAP server, will retry; attempt:8 of 15, delay:5 sec
com.vmware.identity.interop.ldap.ServerDownLdapException: Can't contact LDAP server
LDAP error [code: -1]

thanks,

Justin

[Ajay1988]

I suspect vmdird cert is still expired. Check /usr/lib/vmware-vmdir/share/config/vmdircert.pem  expiry

[i4pcoil]

yes it is... how can I renew/regenerate it? Common Name: vcenter Subject Alternative Names: vcenter Country: US Valid From: April 23, 2019 Valid To: April 22, 2021 Issuer: CA, vcenter Serial Number: e1a2351c764178ff

[Ajay1988]

1. Stop vmdir service: service-control --stop vmdird 2. Backup vmdircert.pem and vmdirkey.pem from: /usr/lib/vmware-vmdir/share/config 3. Easiest is to copy Machine SSL certificate as vmdird certificate: /usr/lib/vmware-vmafd/bin/vecs-cli entry getcert --store machine --alias machine --output /usr/lib/vmware-vmdir/share/config/vmdircert.pem /usr/lib/vmware-vmafd/bin/vecs-cli entry getkey --store machine --alias machine --output /usr/lib/vmware-vmdir/share/config/vmdirkey.pem 4. Restart all services and check

[i4pcoil]

so, I have some progress, all services are up and running now but I still can't connect from

the web client with error:

[400] An error occurred while sending an authentication request to the vCenter Single Sign-On server - An error occurred when processing the metadata during vCenter Single Sign-On setup - Failed to connect to VMware Lookup Service https://vcenter/lookupservice/sdk - SSL certificate verification failed..

If I go to the https://vcenter I get a new certificate which is NOT trusted by the browser and I guess also not trusted by the web client... So what do I need to do on the web client machine to have it connecting properly to the vcenter appliance?

thanks,

Justin

[i4pcoil]

the old C++ GUI gave a certificate warning, I've installed the certificate and it loaded OK, though I threw a warning regarding the update manager

sysimage.fault.SSLCertificateErrror

[Ajay1988]

Nice. So fixing the vmdird cert got things up ?

r u able to  see the inventory on  C++ client ? 

[i4pcoil]

yes, except the update manager option the C++ GUI seems to be working.

But I can't do much there unfortunately and I need to have the HTML5 Web Client working because there is where the juice is done..

Any ideas hot to get the Vcenter certificated trusted by the Web Client appliance?

tnx

[i4pcoil]

ok, system recovered, 3 days VC downtime...

recovered web client by hitting submit on the SSO connection in the web client appliance settings and the system came back online immediately.

I must say the SSL certificates management is a mess and documentation is partial...

thanks to Ajay1988 for pointing me to the vmdird expired certificate, while there is a KB about it it's not linked from other docs and hard to come by.

the invsvc.log doesn't say which certificate is expired, the certificate-manager is NOT generating a certificate for vmdird and somehow only the support staff knows about it..

[Ajay1988]

Good to hear.  Things have changed from 6.5 onwards and   cert replacement is far better and well managed now.

As 6.0 is EOL for more than a year now and you might not find much docs around.  Please resolve this thread 

[VP4]

Hi @Ajay1988 ,

Facing similar issue on windows vCenter 6.0 , could you help with the commands?

[Ajay1988]

@VP4 I hope you have validated that vmdird is the only cert expired. If yes follow below.

1. Stop vmdir service
vCenter Server installed on Windows: service-control --stop VMWareDirectoryService

2. Backup vmdircert.pem and vmdirkey.pem from:
vCenter Server installed on Windows: C:\programdata\vmware\vCenterServer\cfg\vmdird\

3. Copy Machine SSL certificate as vmdird certificate:
C:\Program Files\VMware\vCenter Server\vmafdd\vecs-cli.exe entry getcert --store machine_ssl_cert --alias __machine_cert --output C:\programdata\vmware\vCenterServer\cfg\vmdird\vmdircert.pem

C:\Program Files\VMware\vCenter Server\vmafdd\vecs-cli.exe entry getkey --store machine_ssl_cert --alias __machine_cert --output C:\programdata\vmware\vCenterServer\cfg\vmdird\vmdirkey.pem

4. Start vmdir service
vCenter Server installed on Windows: service-control --start VMWareDirectoryService

[RakeshSWF]

Hi Ajay,

I my case vpxd, vpxd-extension, vsphere-webclient, machine, SMS stores are expired. It is vCenter Server 6 (Windows).

Please share the steps to regenerate SSL certificates as the Certificate Manager is showing 0% Completed [Reset operation failed].

I managed to renew _MACHINE_SSL_CERT, TRUSTED_ROOTS, TRUSTED_ROOT_CRLS.