Hi , I have a VCSA 6.7 with A few days ago, VUM service began to fail and shows the following error, the service is stopped and when I try to start it, the following error is observed
could someone please give me some idea what is causing the failure???, thanks
ERROR TEXT
root@vcenter [ ~ ]# service-control --start vmware-updatemgr
Operation not cancellable. Please wait for it to finish...
Performing start operation on service updatemgr...
Error executing start on service updatemgr. Details {
"detail": [
{
"localized": "An error occurred while starting service 'updatemgr'",
"translatable": "An error occurred while starting service '%(0)s'",
"id": "install.ciscommon.service.failstart",
"args": [
"updatemgr"
]
}
],
"componentKey": null,
"problemId": null,
"resolution": null
}
Service-control failed. Error: {
"detail": [
{
"localized": "An error occurred while starting service 'updatemgr'",
"translatable": "An error occurred while starting service '%(0)s'",
"id": "install.ciscommon.service.failstart",
"args": [
"updatemgr"
]
}
],
"componentKey": null,
"problemId": null,
"resolution": null
}
Did you make any changes to certificates ! Or any recent activity performed !
Hello, export the virtual machine and pass it to another esxi host, after making this migration I do not know if it started to fail
I have this error in vmware-vum-server.log
root@vcenter [ ~ ]# vim /var/log/vmware/vmware-updatemgr/vum-server/vmware-vum-server.log
2019-03-05T11:09:06.052-05:00 info vmware-vum-server[34463] [Originator@6876 sub=ThreadPool] Thread enlisted^M
2019-03-05T11:09:06.052-05:00 info vmware-vum-server[34463] [Originator@6876 sub=ThreadPool] Entering worker thread loop^M
2019-03-05T11:09:40.853-05:00 error vmware-vum-server[33772] [Originator@6876 sub=Ufa.HTTPService] Failed to read request; stream: <SSL(<io_obj p:0x00007fe8c4002a30, h:-1, <TCP '127.0.0.1 : 8084'>, <TCP '127.0.0.1 : 49154'> FD Closed>)>, error: N7Vmacore16TimeoutExceptionE(Operation timed out: Stream: <SSL(<io_obj p:0x00007fe8c4002a30, h:-1, <TCP '127.0.0.1 : 8084'>, <TCP '127.0.0.1 : 49154'> FD Closed>)>, duration: 00:00:45.793412 (hh:mm:ss.us))^M
--> [context]zKq7AVECAQAAAOkXmwALdm13YXJlLXZ1bS1zZXJ2ZXIAAJR/NWxpYnZtYWNvcmUuc28AANQ9HACLShgAhGQbAJhCLADx8CgAYXEoAEV1KAAn8jUB1HMAbGlicHRocmVhZC5zby4wAAKdjA5saWJjLnNvLjYA[/context]^M
2019-03-05T11:17:46.101-05:00 error vmware-vum-server[34463] [Originator@6876 sub=Ufa] Method integrity.VcIntegrity.retrieveVcIntegrityContent threw undeclared fault of type vim.fault.InvalidState^M
2019-03-05T11:18:06.100-05:00 error vmware-vum-server[33771] [Originator@6876 sub=Ufa] Method integrity.VcIntegrity.retrieveVcIntegrityContent threw undeclared fault of type vim.fault.InvalidState^M
2019-03-05T11:18:26.109-05:00 error vmware-vum-server[33823] [Originator@6876 sub=Ufa] Method integrity.VcIntegrity.retrieveVcIntegrityContent threw undeclared fault of type vim.fault.InvalidState^M
2019-03-05T11:18:46.098-05:00 error vmware-vum-server[33820] [Originator@6876 sub=Ufa] Method integrity.VcIntegrity.retrieveVcIntegrityContent threw undeclared fault of type vim.fault.InvalidState^M
2019-03-05T11:19:06.100-05:00 error vmware-vum-server[33770] [Originator@6876 sub=Ufa] Method integrity.VcIntegrity.retrieveVcIntegrityContent threw undeclared fault of type vim.fault.InvalidState^M
2019-03-05T11:19:26.103-05:00 error vmware-vum-server[34389] [Originator@6876 sub=Ufa] Method integrity.VcIntegrity.retrieveVcIntegrityContent threw undeclared fault of type vim.fault.InvalidState^M
2019-03-05T11:19:46.095-05:00 error vmware-vum-server[33772] [Originator@6876 sub=Ufa] Method integrity.VcIntegrity.retrieveVcIntegrityContent threw undeclared fault of type vim.fault.InvalidState^M
2019-03-05T11:20:00.257-05:00 info vmware-vum-server[39217] [Originator@6876 sub=Default] [shutdownMgr,92] Ufa shutdown manager stopping... first make sure this thread is enlisted.^M
2019-03-05T11:20:00.257-05:00 info vmware-vum-server[39217] [Originator@6876 sub=ThreadPool] Thread enlisted^M
2019-03-05T11:20:00.257-05:00 info vmware-vum-server[39217] [Originator@6876 sub=Default] [shutdownMgr,97] Ufa shutdown manager stopping... Total number of callbacks: 1^M
2019-03-05T11:20:00.362-05:00 info vmware-vum-server[39217] [Originator@6876 sub=Default] [shutdownMgr,109] Ufa shutdown manager stopped^M
2019-03-05T11:20:00.362-05:00 info vmware-vum-server[33775] [Originator@6876 sub=ThreadPool] Thread delisted^M
2019-03-05T11:20:00.362-05:00 info vmware-vum-server[33774] [Originator@6876 sub=ThreadPool] Thread delisted^M
2019-03-05T11:20:00.362-05:00 info vmware-vum-server[33773] [Originator@6876 sub=ThreadPool] Thread delisted^M
2019-03-05T11:20:00.362-05:00 info vmware-vum-server[33820] [Originator@6876 sub=ThreadPool] Thread delisted^M
2019-03-05T11:20:00.362-05:00 info vmware-vum-server[33772] [Originator@6876 sub=ThreadPool] Thread delisted^M
2019-03-05T11:20:00.362-05:00 info vmware-vum-server[33823] [Originator@6876 sub=ThreadPool] Thread delisted^M
2019-03-05T11:20:00.362-05:00 info vmware-vum-server[33822] [Originator@6876 sub=ThreadPool] Thread delisted^M
2019-03-05T11:20:00.362-05:00 info vmware-vum-server[33776] [Originator@6876 sub=ThreadPool] Thread delisted^M
2019-03-05T11:20:00.362-05:00 info vmware-vum-server[33770] [Originator@6876 sub=ThreadPool] Thread delisted^M
2019-03-05T11:20:00.362-05:00 info vmware-vum-server[34463] [Originator@6876 sub=ThreadPool] Thread delisted^M
2019-03-05T11:20:00.362-05:00 info vmware-vum-server[33819] [Originator@6876 sub=ThreadPool] Thread delisted^M
2019-03-05T11:20:00.362-05:00 info vmware-vum-server[34389] [Originator@6876 sub=ThreadPool] Thread delisted^M
2019-03-05T11:20:00.362-05:00 info vmware-vum-server[33771] [Originator@6876 sub=ThreadPool] Thread delisted^M
2019-03-05T11:20:01.362-05:00 info vmware-vum-server[33591] [Originator@6876 sub=Ufa] Plugin stopped^M
2019-03-05T11:20:01.362-05:00 info vmware-vum-server[33591] [Originator@6876 sub=ThreadPool] Thread delisted^M
2019-03-05T11:20:01.362-05:00 info vmware-vum-server[33591] [Originator@6876 sub=Default] [main,416] VCI service stopped.^M
Hi,
I'm having the same issue - the VUM service is refusing to start on my vCenter.
We are running with an official SSL certificate for the GUI, to avoid users having to click through SSL errors to access vCenter.
The SSL certificate was able to successfully install in January, so VUM must have started after the installation.
However, we now need to update the SSL certificate again, and can't because VUM is in a failed state (and so any attempt to install a new certificate results in the vCenter rolling back because VUM fails to start).
The problem appears to be that /usr/lib/vmware-updatemgr/bin/refreshCerts-utility.py is failing with a certificate error - from its log file:
----------------
Can't update VUM service info: (vmodl.fault.InvalidArgument) {
dynamicType = <unset>,
dynamicProperty = (vmodl.DynamicProperty) [],
msg = '',
faultCause = <unset>,
faultMessage = (vmodl.LocalizableMessage) [],
invalidProperty = 'Invalid certificate'
}
Traceback (most recent call last):
File "/usr/lib/vmware-updatemgr/bin/vumUtils.py", line 370, in updateServiceInfo
lsClient.ReRegister(vumServiceInfo.serviceId, setSpec)
File "/usr/lib/vmware-updatemgr/bin/vumUtils.py", line 813, in addSecurityctxToRequests
reqMethod(self, *args)
File "/usr/lib/vmware-updatemgr/bin/vumUtils.py", line 823, in ReRegister
sr.Set(svcId, newSpec)
File "/usr/lib/vmware/site-packages/pyVmomi/VmomiSupport.py", line 557, in <lambda>
self.f(*(self.args + (obj,) + args), **kwargs)
File "/usr/lib/vmware/site-packages/pyVmomi/VmomiSupport.py", line 363, in _InvokeMethod
return self._stub.InvokeMethod(self, info, args)
File "/usr/lib/vmware/site-packages/pyVmomi/SoapAdapter.py", line 1418, in InvokeMethod
raise obj # pylint: disable-msg=E0702
pyVmomi.VmomiSupport.vmodl.fault.InvalidArgument: (vmodl.fault.InvalidArgument) {
dynamicType = <unset>,
dynamicProperty = (vmodl.DynamicProperty) [],
msg = '',
faultCause = <unset>,
faultMessage = (vmodl.LocalizableMessage) [],
invalidProperty = 'Invalid certificate'
}
Unable to update VUM service info
----------------
From what little I have been able to work out from the files involved, it looks like it is submitting a certificate from /usr/lib/vmware-updatemgr/bin/ssl/rui.crt to gain access to some service point (I haven't worked out the URL for what it is trying to access), and that is being rejected.
From what little I can see in the vCenter GUI, that certificate file seems to match the certificate for "vpxd-extension" shown in the vCenter administration section.
Is it possible to find logs from the "server" end of the connection, to see why it is having a problem?
Is this someone that is being seen more widely? - it seems to be the same issue as is being described in this thread, which shows that at least some other people are having the problem.
Hi,
I did a bit more digging, and (for my case) it seems that the old certificate was still being retrieved from somewhere (using "LookupServiceClient().GetServiceInfoList()") by vumUtils.py as the "sslTrust" field of the objects it was trying to re-register, and whatever it was trying to register them with was then rejecting the request because that certificate had expired.
I hacked the script to replace that field with the certificate that should have been present (which was already being used by vumUtils.py), and I was then able to get the update service to start, and it seems to be working.
I haven't yet checked to see whether the "LookupServiceClient().GetServiceInfoList()" call is still returning the old certificate, or whether the "re-registration" updated it to the correct one.
Attached is a patch with what I changed, in case it is useful to anyone else.
thank you mfirth this patch fixed my VUM!!
Thanks a ton mfirth, got me out of a pinch.
Hi,
I'm having exactly the same issue in my VCSA 6.5, I did not find vumUtils.py so I'm not sure how can I apply this patch?
Appreciate your help !!
Thank you!
worked in VCSA 6.7.0.42000
I have a similar problem and upgrading the VCSA solves the problem. Can you try this? (provide a full backup of your VCSA, if you decide to upgrade)
vumUtils.patch.zip how to use this patch
Hi mfirth... How can I use your patch?
Hi - I'm not sure if the file is the same or in the same place on 6.5 - the original poster and me are both using 6.7
On 6.7, the file is in /usr/lib/vmware-updatemgr/bin
Hi
If you are using VMware V6.7, then the process is:
Extract vumUtils.patch from the ZIP (on a PC, though it could be done on the VCSA if you really wanted)
Log in to the VCSA and get a root Bash shell (just "shell" if you log in as root, a little more complex if you log in as a non-root user - there must be instructions somewhere on here!)
Cd into /usr/lib/vmware-updatemgr/bin
Copy the patch file into that directory on the VCSA (I did it by using "sftp" out from the VCSA to a Linux machine - there are probably other ways)
Run "patch -p0 < vumUtils.patch" to patch the vumUtils.py file
You should now be able to start the VUM service, either with "service-control --start vmware-updatemgr" from the VMware shell, or by restarting the VCSA
Hi,
See the reply I posted to luckyloveu - hopefully that helps
It looks to me that the certificate is installed right the first time, but then VUM keeps retrieving the original certificate, rather than updating it when it changes.
So a re-install will fix the problem for however long your certificate is valid, but I don't think it is a permanent fix.
So far, I've just kept reapplying my patch whenever I update the VCSA if I have to - it seems to revert it on some updates but not others. I haven't had a problem doing that, so either the patch is still needed or it is at least doing no harm to the newer versions
For every version of VCSA through to 6.7.0.44100 the original vumUtils.py was binary the same as the version I created my patch against, so either VMware still haven't looked into this, or they have fixed it in a different way.
This fixed my issue, thanks! Thankfully I came across this post after already trying multiple "fixes" over several hours. Question: Do you think spinning up a new vCenter, and importing a backup created from the appliance GUI would help solve the issue more permanently?
Many thanks and kudos @mfirth
Worked like a charm and put an end to the hours and hours of finding and trying fixes that didn't make the grade.
Thank you! Fixed it for Version 6.7.0 Build 19832280
Had this issue on a VCSA which got all the expiring certificates renewed recently. The patch worked like a charm.
Thank's a lot @mfirth !
I did it directly on the server
cd /usr/lib/vmware-updatemgr/bin
cp vumUtils.py vumUtils.py.backup
wget https://communities.vmware.com/wbsdv95928/attachments/wbsdv95928/2514/762/1/vumUtils.patch.zip
unzip vumUtils.patch
chown updatemgr:updatemgr ./vumUtils.patch
patch -p0 < vumUtils.patch
It ran fine and showed::
patching file vumUtils.py
Hunk #1 succeeded at 241 (offset 5 lines).
Hunk #2 succeeded at 299 (offset 5 lines).
Hunk #3 succeeded at 344 (offset 5 lines).
Hunk #4 succeeded at 369 (offset 5 lines).
Hunk #5 succeeded at 400 (offset 5 lines).
Hunk #6 succeeded at 409 (offset 5 lines).
Hunk #7 succeeded at 531 (offset 5 lines).
Once again, thank you.