After migrating from Windows 5.5 vCenter to VCSA 6.5 Update1 (intermediate step to 6.7) everything was working well. After the migration the vcenter machine cert was soon to be expired so I replaced it, though I did notice https://vc.domain.com:7444 was still using the old cert. I didn't worry about this as the old cert was not expired. A month or so passes by with vcenter being very stable. Completed host upgrades to 6.5U1 using VUM over the weekend - In the mean time the cert bound to 7444 expired, which caused problems with the Migration wizard to 6.7. Also, VUM service had stopped and wouldn't start (perhaps it was ok until I rebooted the VCSA and now VUM wont start)
With the help of VMWare support we fixed this by replacing the cert in the STS_INTERNAL_SSL_CERT. This fixed the expired cert for 7444 but did not help with the VUM issues. I'm pretty well stuck, I am just waiting on VMWare support to help. In the mean time I thought I'd post here in case someone could possible help.
Regards
I get the following errors in the SSH session when executing service-control --start vmware-updatemgr:
Error executing start on service updatemgr. Details {
"resolution": null,
"detail": [
{
"args": [
"updatemgr"
],
"id": "install.ciscommon.service.failstart",
"localized": "An error occurred while starting service 'updatemgr'",
"translatable": "An error occurred while starting service '%(0)s'"
}
],
"componentKey": null,
"problemId": null
}
Service-control failed. Error {
"resolution": null,
"detail": [
{
"args": [
"updatemgr"
],
"id": "install.ciscommon.service.failstart",
"localized": "An error occurred while starting service 'updatemgr'",
"translatable": "An error occurred while starting service '%(0)s'"
}
],
"componentKey": null,
"problemId": null
}
tail -f updatemgr-utility.log give this:
[2018-10-10 23:29:10,255 INFO] Install Key store for Jetty
[2018-10-10 23:29:11,579 INFO] Keystore installed successfully.
[2018-10-10 23:29:11,953 INFO] Updating VUM extension with VC
[2018-10-10 23:29:12,236 INFO] Updating CM service info
[2018-10-10 23:29:12,402 ERROR] CM ReRegisterService failure. Exception is (cis.cm.fault.ComponentManagerFault) {
dynamicType = <unset>,
dynamicProperty = (vmodl.DynamicProperty) [],
msg = '',
faultCause = <unset>,
faultMessage = (vmodl.LocalizableMessage) [],
errorCode = 0,
errorMessage = 'UNKNOWN'
}
[2018-10-10 23:29:12,402 ERROR] Unable to update CM service info
Tail on cm.log give me this while starting service:
2018-10-10T23:24:35.847Z [pool-2-thread-1 [] WARN com.vmware.cis.services.cm.service.impl.LsVmomiSiteStore (f15657f3-ac72-40f0-8c90-3e948201000c)] Call to lookup service failed; uri:https://VCENTER.DoMain.int/lookupservice/sdk [(vmodl.fault.InvalidArgument) {
faultCause = null,
faultMessage = null,
invalidProperty = Invalid certificate
}]
2018-10-10T23:24:35.847Z [pool-2-thread-1 [] ERROR com.vmware.cis.services.cm.service.ServiceManagerImplTemplate (f15657f3-ac72-40f0-8c90-3e948201000c)] reRegisterService v1: Failed to re-register c39131ca-cda5-446d-a6ac-44b51348c107 (vpxd-extension-a6f84462-5ea2-11e6-ada6-0050569777b5@vsphere.local, com.vmware.vcIntegrity/vcIntegrity 6.5.0)
com.vmware.vim.binding.vmodl.fault.InvalidArgument: null
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) ~[?:1.8.0_141]
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) ~[?:1.8.0_141]
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) ~[?:1.8.0_141]
at java.lang.reflect.Constructor.newInstance(Constructor.java:423) ~[?:1.8.0_141]
at java.lang.Class.newInstance(Class.java:442) ~[?:1.8.0_141]
at com.vmware.vim.vmomi.core.types.impl.ComplexTypeImpl.newInstance(ComplexTypeImpl.java:174) ~[vlsi-core.jar:?]
at com.vmware.vim.vmomi.core.types.impl.DefaultDataObjectFactory.newDataObject(DefaultDataObjectFactory.java:25) ~[vlsi-core.jar:?]
at com.vmware.vim.vmomi.core.soap.impl.unmarshaller.ComplexStackContext.<init>(ComplexStackContext.java:30) ~[vlsi-core.jar:?]
at com.vmware.vim.vmomi.core.soap.impl.unmarshaller.UnmarshallerImpl$UnmarshallSoapFaultContext.parse(UnmarshallerImpl.java:150) ~[vlsi-core.jar:?]
at com.vmware.vim.vmomi.core.soap.impl.unmarshaller.UnmarshallerImpl$UnmarshallSoapFaultContext.unmarshall(UnmarshallerImpl.java:101) ~[vlsi-core.jar:?]
at com.vmware.vim.vmomi.core.soap.impl.unmarshaller.UnmarshallerImpl.unmarshalSoapFault(UnmarshallerImpl.java:88) ~[vlsi-core.jar:?]
at com.vmware.vim.vmomi.core.soap.impl.unmarshaller.UnmarshallerImpl.unmarshalSoapFault(UnmarshallerImpl.java:83) ~[vlsi-core.jar:?]
at com.vmware.vim.vmomi.client.common.impl.SoapFaultStackContext.setValue(SoapFaultStackContext.java:40) ~[vlsi-client.jar:?]
at com.vmware.vim.vmomi.client.common.impl.ResponseUnmarshaller.processNextElement(ResponseUnmarshaller.java:127) ~[vlsi-client.jar:?]
at com.vmware.vim.vmomi.client.common.impl.ResponseUnmarshaller.unmarshal(ResponseUnmarshaller.java:70) ~[vlsi-client.jar:?]
at com.vmware.vim.vmomi.client.common.impl.ResponseImpl.unmarshalResponse(ResponseImpl.java:274) ~[vlsi-client.jar:?]
at com.vmware.vim.vmomi.client.common.impl.ResponseImpl.setResponse(ResponseImpl.java:230) ~[vlsi-client.jar:?]
at com.vmware.vim.vmomi.client.http.impl.HttpExchangeBase.parseResponse(HttpExchangeBase.java:150) ~[vlsi-client.jar:?]
at com.vmware.vim.vmomi.client.http.impl.HttpExchange.run(HttpExchange.java:48) ~[vlsi-client.jar:?]
at com.vmware.vim.vmomi.client.http.impl.HttpProtocolBindingBase.executeRunnable(HttpProtocolBindingBase.java:226) ~[vlsi-client.jar:?]
at com.vmware.vim.vmomi.client.http.impl.HttpProtocolBindingImpl.send(HttpProtocolBindingImpl.java:110) ~[vlsi-client.jar:?]
at com.vmware.vim.vmomi.client.common.impl.MethodInvocationHandlerImpl$CallExecutor.sendCall(MethodInvocationHandlerImpl.java:613) ~[vlsi-client.jar:?]
at com.vmware.vim.vmomi.client.common.impl.MethodInvocationHandlerImpl$CallExecutor.executeCall(MethodInvocationHandlerImpl.java:594) ~[vlsi-client.jar:?]
at com.vmware.vim.vmomi.client.common.impl.MethodInvocationHandlerImpl.completeCall(MethodInvocationHandlerImpl.java:345) ~[vlsi-client.jar:?]
at com.vmware.vim.vmomi.client.common.impl.MethodInvocationHandlerImpl.invokeOperation(MethodInvocationHandlerImpl.java:305) ~[vlsi-client.jar:?]
at com.vmware.vim.vmomi.client.common.impl.MethodInvocationHandlerImpl.invoke(MethodInvocationHandlerImpl.java:179) ~[vlsi-client.jar:?]
at com.sun.proxy.$Proxy101.set(Unknown Source) ~[?:?]
at com.vmware.cis.services.cm.service.impl.LsVmomiSiteStore$LsVmomiWrapper$3.execute(LsVmomiSiteStore.java:229) ~[service-cm.jar:?]
at com.vmware.cis.services.cm.service.impl.LsVmomiSiteStore$LsVmomiWrapper$3.execute(LsVmomiSiteStore.java:226) ~[service-cm.jar:?]
at com.vmware.cis.services.cm.service.impl.LsVmomiSiteStore$LsVmomiWrapper.callLs(LsVmomiSiteStore.java:302) ~[service-cm.jar:?]
at com.vmware.cis.services.cm.service.impl.LsVmomiSiteStore$LsVmomiWrapper.set(LsVmomiSiteStore.java:224) ~[service-cm.jar:?]
at com.vmware.cis.services.cm.service.impl.LsVmomiSiteStore.updateService(LsVmomiSiteStore.java:622) ~[service-cm.jar:?]
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_141]
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:1.8.0_141]
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_141]
at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_141]
at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:333) ~[spring-aop-4.3.9.RELEASE.jar:4.3.9.RELEASE]
at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:190) ~[spring-aop-4.3.9.RELEASE.jar:4.3.9.RELEASE]
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:157) ~[spring-aop-4.3.9.RELEASE.jar:4.3.9.RELEASE]
at com.vmware.cis.services.common.perfmon.PerfmonInterceptor.invoke(PerfmonInterceptor.java:31) ~[service-common.jar:?]
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179) ~[spring-aop-4.3.9.RELEASE.jar:4.3.9.RELEASE]
at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:92) ~[spring-aop-4.3.9.RELEASE.jar:4.3.9.RELEASE]
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179) ~[spring-aop-4.3.9.RELEASE.jar:4.3.9.RELEASE]
at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:213) ~[spring-aop-4.3.9.RELEASE.jar:4.3.9.RELEASE]
at com.sun.proxy.$Proxy67.updateService(Unknown Source) ~[?:?]
at com.vmware.cis.services.cm.service.ServiceManagerImplTemplate.reRegisterService(ServiceManagerImplTemplate.java:306) [service-cm.jar:?]
at com.vmware.cis.services.cm.service.ServiceManagerImpl.reRegisterService(ServiceManagerImpl.java:291) [service-cm.jar:?]
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_141]
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:1.8.0_141]
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_141]
at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_141]
at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:333) [spring-aop-4.3.9.RELEASE.jar:4.3.9.RELEASE]
at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:190) [spring-aop-4.3.9.RELEASE.jar:4.3.9.RELEASE]
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:157) [spring-aop-4.3.9.RELEASE.jar:4.3.9.RELEASE]
at com.vmware.cis.services.common.perfmon.PerfmonInterceptor.invoke(PerfmonInterceptor.java:31) [service-common.jar:?]
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179) [spring-aop-4.3.9.RELEASE.jar:4.3.9.RELEASE]
at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:92) [spring-aop-4.3.9.RELEASE.jar:4.3.9.RELEASE]
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179) [spring-aop-4.3.9.RELEASE.jar:4.3.9.RELEASE]
at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:213) [spring-aop-4.3.9.RELEASE.jar:4.3.9.RELEASE]
at com.sun.proxy.$Proxy68.reRegisterService(Unknown Source) [?:?]
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_141]
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:1.8.0_141]
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_141]
at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_141]
at com.vmware.vim.vmomi.server.impl.InvocationTask.run(InvocationTask.java:65) [vlsi-server.jar:?]
at com.vmware.vim.vmomi.server.common.impl.RunnableWrapper$1.run(RunnableWrapper.java:47) [vlsi-server.jar:?]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_141]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_141]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_141]
tcpdump on lo gives:
Request
POST /lookupservice/sdk HTTP/1.1
Content-Type: text/xml; charset=utf-8
SOAPAction: urn:lookup/2.0
Content-Length: 21474
Host: https://VCENTER.DoMain.int/lookupservice/sdk
Connection: Keep-Alive
User-Agent: VMware vim-java 1.0
Cookie: vmware_soap_session=a7600162-79b1-4797-9e7f-4b885dde550b
Accept-Encoding: gzip,deflate
X-Forwarded-For: 127.0.0.1
X-Forwarded-Proto: https
....CERT HASH in XML stream.........
Request
HTTP/1.1 500
Set-Cookie: vmware_soap_session=a7600162-79b1-4797-9e7f-4b885dde550b; HttpOnly
Content-Type: text/xml;charset=utf-8
Content-Length: 558
Date: Wed, 10 Oct 2018 01:45:27 GMT
Connection: close
<?xml version='1.0' encoding='UTF-8'?><soapenv:Envelope xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/" xmlns:soapenc="http://schemas.xmlsoap.org/soap/encoding/" xmlns:xsd="http://www.w3.org/2001/XMLSchema" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"><soapenv:Body><soapenv:Fault><faultcode>ServerFaultCode</faultcode><faultstring/><detail><RuntimeFaultFault xsi:type="InvalidArgument" xmlns="urn:lookup"><invalidProperty>Invalid certificate</invalidProperty></RuntimeFaultFault></detail></soapenv:Fault></soapenv:Body></soapenv:Envelope>
ssh into vcsa
cd /usr/lib/vmware-updatemgr/bin and make sure configvalues.txt and vci-integrity.xml files aren't empty.
Thanks for the replies. Vmware got in there yesterday and sorted. The endpoints for VUM were trusting the wrong cert. They updated the ssltrusts with the correct cert hash and all good now.
You will have to use ls_update_certs.py script to correct this to correct thumbprint mismatch.
I would suggest you to log a case with GSS.
In case someone stumbles here with the similar problem, the process described in this KB helped solve this problem for me: https://kb.vmware.com/s/article/2121689
(Don't forget to backup your vcenter before proceeding)
In short, the process to fix the thumbprint mismatch is as follows:
1) get the current Machine SSL certificate:
/usr/lib/vmware-vmafd/bin/vecs-cli entry getcert --store MACHINE_SSL_CERT --alias __MACHINE_CERT --output /tmp/machineSSL.crt
2) Get the SSL trust:
/usr/lib/vmidentity/tools/scripts/lstool.py list --url https://localhost/lookupservice/sdk --no-check-cert --ep-type com.vmware.cis.cs.identity.sso 2>/dev/null
3) Create some /tmp/newcert.crt with the content like:
-----BEGIN CERTIFICATE-----
<Insert contents of "SSL trust" field here>
-----END CERTIFICATE-----
4) get the thumbprints:
openssl x509 -in /tmp/machineSSL.crt -fingerprint -noout
openssl x509 -in /tmp/newcert.crt -fingerprint -noout
you can confirm that this is your issue here. If the thumbprints do not match, than this is indeed your issue and you need to do the following:
5)Run this script
/usr/lib/vmidentity/tools/scripts/ls_update_certs.py --url http://localhost:7080/lookupservice/sdk --fingerprint <insert fingerprint from newcert.crt> --certfile /tmp/machineSSL.crt --user administrator@vsphere.local --password <password>
Hope this helps.
Thanks for the information,
Kukezh, that finally made my day after ton´s of senseless articles that all failed to gegenerate the VUM Certs !
Amazing, this helped me too to solve a not starting update-manager service on a VCSA 6.7 U3.
THANK YOU!
Thank you Kukezh
it worked for me.
comment:
if you have special characters in your password tyte it in single quotes; in my case my password was Passw0rd$$ . below command i used.
/usr/lib/vmidentity/tools/scripts/ls_update_certs.py --url http://localhost:7080/lookupservice/sdk --fingerprint 7B:B8:AC:F8:1F:D6:26:7C:3B:FD:24:0D:92:42:44:61:E8:CF:2E:A8 --certfile /tmp/machineSSL.crt --user administrator@vsphere.local --password 'Passw0rd$$'
Hello Kukezh,
I have a VCSA 6.5 with external PSCs. That command returns nothing on the VCSA itself:
/usr/lib/vmidentity/tools/scripts/lstool.py list --url https://localhost/lookupservice/sdk --no-check-cert --ep-type com.vmware.cis.cs.identity.sso 2>/dev/null
It does return something on my two PSCs. I can see the SSL Trust for my VCSA's hostname in the list.
Can i still use your fix?
Hi,
I have no experience with external PSC, but as PSC is the one who controls certificates, I assume you need to run this on the PSC itself, though I'm not sure.
Sorry, that I'm not much of a help here.
After 3 days looking for a solution I found this, and it worked as expected. I believe you should include this to the KB 76298 VMware Knowledge Base
Thanks,
Marcelo
Thank you so much, Kukezh! I had this same issue and this solution here worked (after I tried several others all afternoon).
Thanks a lot Kukezh. my vCenter is in 5.5 but this procedure work just fine.
Solution still working 😉
Thanks a lot for the clear description!