Hello everyone!
I`ve got two vcenter servers(vcenter1 and vcenter2) with external PSC(psc1 and psc2). The problem is that on vcenter2 certificates were updated but on psc havent added new certificates to active. So if i log in vcenter2 i can manage vcenter2 and vcenter1, but if i log into vcenter1 i see message "Cannot connect to one or more vcenter servers".
I`ve googled for case like this, but could not find anything. Can anyone point me how to solve this problem?
Moderator: Moved to vCenter Server Discussions
when you login to VC1, we need to check the webclient logs for the vc1 and see why its not able to communicate with vc2.
do you have all ports opened with psc/vcs in your environment? Is the behavior same with SSO administrator as well or only with domain accounts?
thanks,
MS
Hello msripada,
there are no errors in web client logs, but in /var/log/vmware/vapi/endpoint/endpoint.log is see errors like this:
com.vmware.vim.query.client.exception.ClientException: java.util.concurrent.ExecutionException: com.vmware.vim.vmomi.client.exception.SslException: com.vmware.vim.vmomi.core.exception.CertificateValidationException: Server certificate chain is not trusted and thumbprint verification is not configured
at com.vmware.vim.query.client.impl.QueryAuthenticationManagerImpl.loginBySamlToken(QueryAuthenticationManagerImpl.java:232)
at com.vmware.vapi.endpoint.cis.router.InvProviderClientFactory.createProviderClient(InvProviderClientFactory.java:105)
at com.vmware.vapi.endpoint.cis.router.InvSvcBuilder.createInvServiceClientList(InvSvcBuilder.java:345)
at com.vmware.vapi.endpoint.cis.router.InvSvcBuilder.buildInt(InvSvcBuilder.java:296)
at com.vmware.vapi.endpoint.cis.router.InvSvcBuilder.rebuild(InvSvcBuilder.java:254)
at com.vmware.vapi.state.impl.DefaultStateManager.rebuild(DefaultStateManager.java:406)
at com.vmware.vapi.state.impl.DefaultStateManager$2.doReconfig(DefaultStateManager.java:444)
at com.vmware.vapi.state.impl.DefaultStateManager$2.run(DefaultStateManager.java:433)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.util.concurrent.ExecutionException: com.vmware.vim.vmomi.client.exception.SslException: com.vmware.vim.vmomi.core.exception.CertificateValidationException: Server certificate chain is not trusted and thumbprint verification is not configured
at com.vmware.vim.vmomi.core.impl.BlockingFuture.get(BlockingFuture.java:81)
at com.vmware.vim.query.client.impl.QueryAuthenticationManagerImpl.loginBySamlToken(QueryAuthenticationManagerImpl.java:230)
... 14 more
Caused by: com.vmware.vim.vmomi.client.exception.SslException: com.vmware.vim.vmomi.core.exception.CertificateValidationException: Server certificate chain is not trusted and thumbprint verification is not configured
at com.vmware.vim.vmomi.client.common.impl.ResponseImpl.setError(ResponseImpl.java:256)
at com.vmware.vim.vmomi.client.http.impl.HttpExchange.run(HttpExchange.java:51)
at com.vmware.vim.vmomi.client.http.impl.HttpProtocolBindingBase.executeRunnable(HttpProtocolBindingBase.java:226)
at com.vmware.vim.vmomi.client.http.impl.HttpProtocolBindingImpl.send(HttpProtocolBindingImpl.java:110)
at com.vmware.vim.vmomi.client.common.impl.MethodInvocationHandlerImpl$CallExecutor.sendCall(MethodInvocationHandlerImpl.java:613)
at com.vmware.vim.vmomi.client.common.impl.MethodInvocationHandlerImpl$CallExecutor.executeCall(MethodInvocationHandlerImpl.java:594)
at com.vmware.vim.vmomi.client.common.impl.MethodInvocationHandlerImpl.completeCall(MethodInvocationHandlerImpl.java:345)
at com.vmware.vim.vmomi.client.common.impl.MethodInvocationHandlerImpl.invokeOperation(MethodInvocationHandlerImpl.java:305)
at com.vmware.vim.vmomi.client.common.impl.MethodInvocationHandlerImpl.invoke(MethodInvocationHandlerImpl.java:179)
at com.sun.proxy.$Proxy91.loginBySamlToken(Unknown Source)
at com.vmware.vim.query.client.impl.QueryAuthenticationManagerImpl.loginBySamlToken(QueryAuthenticationManagerImpl.java:228)
... 14 more
Caused by: com.vmware.vim.vmomi.core.exception.CertificateValidationException: Server certificate chain is not trusted and thumbprint verification is not configured
at com.vmware.vim.vmomi.client.http.impl.ClientExceptionTranslator.translate(ClientExceptionTranslator.java:54)
... 25 more
Caused by: com.vmware.vim.vmomi.core.exception.CertificateValidationException: SSL handshake from 0.0.0.0/0.0.0.0:53206 to vcenter2/172.22.0.253:443 failed in 25 ms
at com.vmware.vim.vmomi.client.http.impl.ThumbprintTrustManager$HostnameVerifier.handleHandshakeException(ThumbprintTrustManager.java:597)
at com.vmware.vim.vmomi.client.http.impl.ThumbprintTrustManager$HostnameVerifier.verify(ThumbprintTrustManager.java:422)
at com.vmware.vim.vmomi.client.http.impl.VlsiSslSocketFactory.verifyHostname(VlsiSslSocketFactory.java:129)
at com.vmware.vim.vmomi.client.http.impl.VlsiSslSocketFactory.createLayeredSocket(VlsiSslSocketFactory.java:122)
at com.vmware.vim.vmomi.client.http.impl.VlsiSslSocketFactory.connectSocket(VlsiSslSocketFactory.java:88)
at org.apache.http.impl.conn.HttpClientConnectionOperator.connect(HttpClientConnectionOperator.java:117)
at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:314)
at org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:363)
at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:219)
at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:195)
at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:86)
at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:108)
at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:186)
at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:82)
at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:57)
at com.vmware.vim.vmomi.client.http.impl.HttpExchange.run(HttpExchange.java:45)
... 23 more
Caused by: javax.net.ssl.SSLHandshakeException: com.vmware.vim.vmomi.client.exception.VlsiCertificateException: Server certificate chain is not trusted and thumbprint verification is not configured
at sun.security.ssl.Alerts.getSSLException(Alerts.java:198)
at sun.security.ssl.SSLSocketImpl.fatal(SSLSocketImpl.java:1967)
at sun.security.ssl.Handshaker.fatalSE(Handshaker.java:331)
at sun.security.ssl.Handshaker.fatalSE(Handshaker.java:325)
at sun.security.ssl.ClientHandshaker.serverCertificate(ClientHandshaker.java:1689)
at sun.security.ssl.ClientHandshaker.processMessage(ClientHandshaker.java:226)
at sun.security.ssl.Handshaker.processLoop(Handshaker.java:1082)
at sun.security.ssl.Handshaker.process_record(Handshaker.java:1010)
at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:1079)
at sun.security.ssl.SSLSocketImpl.performInitialHandshake(SSLSocketImpl.java:1388)
at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1416)
at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1400)
at com.vmware.vim.vmomi.client.http.impl.ThumbprintTrustManager$HostnameVerifier.verify(ThumbprintTrustManager.java:420)
... 37 more
Caused by: com.vmware.vim.vmomi.client.exception.VlsiCertificateException: Server certificate chain is not trusted and thumbprint verification is not configured
at com.vmware.vim.vmomi.client.http.impl.ThumbprintTrustManager.checkServerTrusted(ThumbprintTrustManager.java:206)
at sun.security.ssl.AbstractTrustManagerWrapper.checkServerTrusted(SSLContextImpl.java:1099)
at sun.security.ssl.ClientHandshaker.serverCertificate(ClientHandshaker.java:1671)
... 45 more
Caused by: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
at sun.security.validator.PKIXValidator.doBuild(PKIXValidator.java:450)
at sun.security.validator.PKIXValidator.engineValidate(PKIXValidator.java:317)
at sun.security.validator.Validator.validate(Validator.java:262)
at sun.security.ssl.X509TrustManagerImpl.validate(X509TrustManagerImpl.java:330)
at sun.security.ssl.X509TrustManagerImpl.checkTrusted(X509TrustManagerImpl.java:235)
at sun.security.ssl.X509TrustManagerImpl.checkServerTrusted(X509TrustManagerImpl.java:113)
at com.vmware.vim.vmomi.client.http.impl.ThumbprintTrustManager.checkServerTrusted(ThumbprintTrustManager.java:191)
... 47 more
Caused by: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
at sun.security.provider.certpath.SunCertPathBuilder.build(SunCertPathBuilder.java:141)
at sun.security.provider.certpath.SunCertPathBuilder.engineBuild(SunCertPathBuilder.java:126)
at java.security.cert.CertPathBuilder.build(CertPathBuilder.java:280)
at sun.security.validator.PKIXValidator.doBuild(PKIXValidator.java:445)
... 53 more
MACHINE SSL of vcenter 2 is having issues with trust mismatch. You can use lsdoctor https://kb.vmware.com/s/article/80469 but you need to have maintenance to shutdown and take powered off snapshots of all vcenter/pscs in the environment. Use lsdoctor -t once you have snaps and backups ready.
thanks,
MS
Moderator: Please use the "spoiler" function when posting large text dumps to make the thread readable by others, I have edited your most recent post so you can see the difference.
You add a "spoiler" to a post using the triangle icon on the extended toolbar of the post creator/editor:
Thanks for answer.
i`ve tried lsdoctor util, but got this:
But vcenter2 appeared in web client of vcenter1, but still not managebale.
Is ther a way to upload new certificates of vcenter2 to psc1?
I suggest you to kindly open case with GSS as things may get complicated if we tweak issues with certs
@smelnik were you able to resolve this issue. If so please help us with the resolution. We are also seeing this issue in our environment.