Running VCSA 6.7.0.42300 and attempting to upgrade to 7.0.0.10100. The new VCSA is deployed fine, and during stage 2 right after the pre-upgrade tests are run I receive the following error.
Internal error occurs during execution of upgrade process.
Please search for these symptoms in the VMware Knowledge Base for any known issues and possible resolutions. If none can be found, collect a support bundle and open a support request.
I reviewed the logs and in the "requirements-upgrade-runner.log" the last error is this:
2020-04-03T05:27:57.786Z INFO utils.loginspec_utils User credentials based login specification.
2020-04-03T05:27:57.787Z INFO utils.loginspec_utils The vCenter Server username and password are specified.
2020-04-03T05:27:57.804Z INFO bindings.vc_binding User credentials based authentication.
2020-04-03T05:27:57.863Z INFO deployer.migration_env_deployer Updating extension -- com.vmware.migrate-connector.127.0.0.1
2020-04-03T05:27:57.932Z ERROR UpgradeRunner Upgrade Runner has encountered an exception
Traceback (most recent call last):
File "/tmp/vmware-upgrade-temp-dirdvIaAnB9OK/tmp8khvlEWIMa/UpgradeRunner.py", line 1889, in main
envDeployer.setup(srcTargets, destTargets, isRegVcExt)
File "/tmp/vmware-upgrade-temp-dirdvIaAnB9OK/tmp8khvlEWIMa/py/deployer/migration_env_deployer.py", line 563, in setup
self._setupTargets(srcTargetsToSetup, False, isRegVcExt)
File "/tmp/vmware-upgrade-temp-dirdvIaAnB9OK/tmp8khvlEWIMa/py/deployer/migration_env_deployer.py", line 535, in _setupTargets
self._registerTargetAsExtension(targetConfig)
File "/tmp/vmware-upgrade-temp-dirdvIaAnB9OK/tmp8khvlEWIMa/py/deployer/migration_env_deployer.py", line 345, in _registerTargetAsExtension
_registerExtension(srcSi, extKey, extCertificate)
File "/tmp/vmware-upgrade-temp-dirdvIaAnB9OK/tmp8khvlEWIMa/py/deployer/migration_env_deployer.py", line 78, in _registerExtension
em.UpdateExtension(ext)
File "/tmp/vmware-upgrade-temp-dirdvIaAnB9OK/tmp8khvlEWIMa/libs/pyVmomi.zip/pyVmomi/VmomiSupport.py", line 556, in <lambda>
self.f(*(self.args + (obj,) + args), **kwargs)
File "/tmp/vmware-upgrade-temp-dirdvIaAnB9OK/tmp8khvlEWIMa/libs/pyVmomi.zip/pyVmomi/VmomiSupport.py", line 368, in _InvokeMethod
return self._stub.InvokeMethod(self, info, args)
File "/tmp/vmware-upgrade-temp-dirdvIaAnB9OK/tmp8khvlEWIMa/libs/pyVmomi.zip/pyVmomi/SoapAdapter.py", line 1481, in InvokeMethod
raise obj # pylint: disable-msg=E0702
pyVmomi.VmomiSupport.vmodl.fault.SystemError: (vmodl.fault.SystemError) {
dynamicType = <unset>,
dynamicProperty = (vmodl.DynamicProperty) [],
msg = 'lookup.fault.EntryNotFoundFault',
faultCause = <unset>,
faultMessage = (vmodl.LocalizableMessage) [],
reason = 'Invalid fault'
}
Please advise on how to further troubleshoot the upgrade.
I was finally able to pass the pre-upgrade checks!
First I had to remove the "com.vmware.migrate-connector.127.0.0.1" extension in order to resolve the "Internal error occurs during execution of upgrade process" error. To do this I had to:
I then restarted the vCenter 6.7 VCSA and ran stage 2 of the upgrade again. This time upon completing the pre-upgrade checks I received a different error: "Certificate validation failed during pre-upgrade check". I reset all certs (/usr/lib/vmware-vmca/bin/certificate-manager" option 8 ), but this itself did not fix anything, so that step is likely optional. I then SSH'd into the VCSA and ran the following the commands as instructed in the this KB article: VMware Knowledge Base
I then restarted the VCSA again, ran stage 2 upgrade and it passed the checks and successfully upgraded to 7.0.
Many thanks to tomosimeonov for assisting in this problem!
i think you should open a support request.
vpshere 7.0 was released yesterday so you will find no one here who has experience with 7.0
Hi bascheew,
Can you please provide some information about your source VCSA 6.7 env ? whether any solutions or extension registered to it ? Is it possible to share the requirements-upgrade-runner.log ?
Regards
We don't have any extensions that I'm aware of using vCenter. The environment is fairly simple - one Datacenter, 3 Clusters, a couple hundred VMs, DRS, distributed switches, SAN storage. We originally started with vCenter 5 on Windows and have upgraded it over the years to VCSA 6.7. I wonder if there is some old garbage lingering around from previous versions.
Attached is a sanitized log file. Names and IPs have been changed.
I will add that the first time we ran the prerequisite check I had 2 errors - one was cert related so I renewed all the certs on the old VCSA and that error went away. The second error was about vswitches and folders having the same name. I just renamed the switches or folders and then that error went away -- leaving me with the error I've posted about.
The following article seems awfully close to the problem I'm having: https://kb.vmware.com/s/article/78296?lang=en_US%3Flang%3Den_US
However I don't see any VMs showing up as "invalid" when I run the following command across all my hosts.
vim-cmd vmsvc/getallvms
I'm not sure this is the same issue...
My issue has thrown the same exception at the same place. I'm clueless at this point, I've been hammering on it for a few hours, exploring every avenue.
Thanks . Can you please share the Upgraderunner.log as well?
Hi,
I think you might have stale entries in your ExtensionManager on the 6.7 environment. Can you check if you have Extension called com.vmware.migrate-connector.127.0.0.1 and if so remove it. If that does not solve your problem I would highly recommend raising a support request.
Thanks,
Tomo
https://vcenter.domain.local/mob/?moid=ExtensionManager I did find com.vmware.migrate-connector.127.0.0.1 there and was able to remove it. As the other person had problems with Certs, so did I. They seemingly went away... After unregistering that extension, I had the same cert problem... I'm sure it didn't help that I probably tried the upgrade 10+ times last night... Okay:
"A vCenter Single Sign-On endpoint certificate validation error has occurred. Resolution Ensure that the endpoint service registrations in vmdir match their corrsponding machine SSL certificates in VECS. For more information, see Knowledge Base article KB 2121701"
I was able to get the "old cert" and run the ls_update_certs.py script. Now it seems to be working fine. What a nightmare, hopefully this helps somebody.
*To clarify, I'm running a 6.7 vcsa with embedded psc*
HI, jmhalder,
Now what's the state of upgrade now. are you able to proceed with stage2.
Correct, stage 2 completed, and I'm upgraded to 7.0 now.
jmhalder thanks for confirmation.
Can you share more information regarding the history of VC.
you have told it started from 5.0. we want to know what all path you traversed to reach 67 like (5.0->5.1->5.5->6.7/5,0->5,1->5.5->6.0->6.5->6.7 etc) . where did you do the migration etc.
Thank you for your assistance!
It's been a long time, but I believe the upgrade path was 5.0 Windows -> 5.1 Windows -> 5.5 Windows -> 6.0 VCSA -> 6.5 VCSA -> 6.7 VCSA.
I found that I also had the extension "com.vmware.migrate-connector.127.0.0.1" installed so I removed it. Now I am getting a cert error instead:
Error: Certificate validation failed during pre-upgrade check.
Resolution: Please contact support
I will start troubleshooting this error and report back on my findings.
The requirement-upgrade-runner.log contain the following line:
INFO ssoCollectValidate certDetails {} , certMismatch {'resolution': Please contact support, 'problemId': None, 'description': Unexpected error 87 during certificate pre-check., 'severity': 'ERROR', 'text': Certificate validation failed during pre-upgrade check.}
Under "Certificate Management" of vCenter all certs show a green check and are valid. This error came from the ssoCollectVaildate stage, which would indicate an SSO cert perhaps? Any idea where I should look next?
I ran the certificate manager on the 6.7 vCenter CLI "/usr/lib/vmware-vmca/bin/certificate-manager" and chose option 8 - Reset all Certificates. After that command successfully ran, I rebooted the vCenter server and restarted stage 2 of the VCSA 7.0 upgrade. I still receive the same pre-upgrade certificate error. Any other suggestions on what I can do?
I was finally able to pass the pre-upgrade checks!
First I had to remove the "com.vmware.migrate-connector.127.0.0.1" extension in order to resolve the "Internal error occurs during execution of upgrade process" error. To do this I had to:
I then restarted the vCenter 6.7 VCSA and ran stage 2 of the upgrade again. This time upon completing the pre-upgrade checks I received a different error: "Certificate validation failed during pre-upgrade check". I reset all certs (/usr/lib/vmware-vmca/bin/certificate-manager" option 8 ), but this itself did not fix anything, so that step is likely optional. I then SSH'd into the VCSA and ran the following the commands as instructed in the this KB article: VMware Knowledge Base
I then restarted the VCSA again, ran stage 2 upgrade and it passed the checks and successfully upgraded to 7.0.
Many thanks to tomosimeonov for assisting in this problem!
Hello,
Me too. Somebody have some solution?
Thanks
I login first time to the community with my account (as far as i remember) just to say thank you to you bascheew
I worked since the morning on updating the vcsa and already fixed some else problems. But this last one i could not figure out. Your hint was the final solution i needed. Thank you VERY MUCH!
amazing. I finally got it to work and I've been on this for 3 days now.
I think the important part for us was to stop and restart the services with
service-control --stop --all
service-control --start --all
or just reboot the appliance.
FYI: we have an appliance running 6.7 and upgrading to 7.0.
I tried initially to go from 6.5 to 6.7 when making the jump straight to 7.0 gave the 87 error, but even with the interim upgrade, it still failed. ultimately I think the cert manager option 8, restarting services, then running the commands listed in the solution worked!
thanks for using time to add the info here and help others.
I had the same problem upgrading from 7.0 to 8. The solution worked in my case, also. I did not restart the source VCSA nor restarted the services after removing the com.vmware.migrate-connector (I did not touch the certificates.
Thanks again for sharing