503 Service Unavailable (Failed to connect to endpoint: [class Vmacore::Http::NamedPipeServiceSpec:0x00000287f89b5010] _serverNamespace = / action = Allow _pipeName =\\.\pipe\vmware-vpxd-webserver-pipe)

MaydayIT · ‎04-15-2019

Hello everyone.

I've been getting 503 errors trying to connect to our VC6.5 Windows-based server. The environment is a three-host cluster running a learning environment. I know the first recommendation is going to be upgrading to a 6.7 appliance, which is valid, but I can't upgrade in place because of domain restrictions and changing the hostname of the VCenter server would break the appliance that creates the labs. The exact error I'm seeing is

503 Service Unavailable (Failed to connect to endpoint: [class Vmacore::Http::NamedPipeServiceSpec:0x00000287f89b5010] _serverNamespace = / action = Allow _pipeName =\\.\pipe\vmware-vpxd-webserver-pipe)

Any idea what might be causing this? I'll attach a log bundle in case it helps.

Thanks for any help you can offer.

Derek

msripada · ‎04-24-2019

Hello MaydayIT,

Thanks for the logs. It is the space on the vcenter database location which is full..

2019-04-18T10:29:45.639-06:00 info vpxd[10404] [Originator@6876 sub=CryptoManager] KMS cert earlist expiry time is 2147483647, check interval is 43200000000

2019-04-18T10:29:45.719-06:00 error vpxd[10452] [Originator@6876 sub=vpxdVdb] Shutting down the VC as there is not enough free space for the Database(used: 99%; threshold: 95%).

2019-04-18T10:29:45.720-06:00 info vpxd[10452] [Originator@6876 sub=Default] Initiating VMware VirtualCenter shutdown

2019-04-18T10:29:45.921-06:00 info vpxd[09968] [Originator@6876 sub=vpxCrypt] Failed to read X509 cert; err: 151441516

vCenter 6.5/6.7 has added capability which will shutdown vcenter service when the space is at 95% used by design to avoid database corruptions.

Can you clear some space/increase which should bring the vCenter up and running

Thanks,

MS

View solution in original post

MikeStoica · ‎04-16-2019

Did you just rebooted your vCenter?

MaydayIT · ‎04-16-2019

Thanks for your reply. I had rebooted it a few hours before, but not that recently.

diegodco31 · ‎04-16-2019

Hi

Check if the following VMware KB article helps:

VMware Knowledge Base

If not work, attach the log file.

Windows vCenter Server: C:\ProgramData\VMware\vCenterServer\logs\vmware-vpx\vpxd.log

Diego Oliveira
LinkedIn: http://www.linkedin.com/in/dcodiego

MaydayIT · ‎04-16-2019

Thanks, but nothing in the linked list appears to fit my situation. This was a clean install of 6.5. It's not an upgrade, not an appliance, and the database is not local. I've had this happen in the past, and a rebuild of VCenter would normally last at least a few months. This time, it crapped out again after about six weeks and I just can't keep rebuilding this beast!

The log bundle is attached to the first message.

Derek

msripada · ‎04-16-2019

Can you check this if its windows vcenter?

VMware Knowledge Base

Secondary, please run the command via opening a cmd (as administrator

cd c:\program files\vmware\vcenterserver\bin

service-control --status --all

Thanks,

MS

MaydayIT · ‎04-16-2019

Thanks Msripada. Here's the output from the

---

Running:

EsxAgentManager VMWareAfdService VMWareCertificateService VMWareDirectoryService VMwareComponentManager VMwareDNSServic

e VMwareIdentityMgmtService VMwareSTS VServiceManager content-library rhttpproxy vapiEndpoint vimPBSM vmon vmonapi vmsys

logcollector vmware-cis-config vmware-license vmware-perfcharts vmware-psc-client vmwareServiceControlAgent vpxd-svcs vs

an-health vsphere-ui vspherewebclientsvc

Stopped:

VMWareCAMService mbcs vmware-autodeploy-waiter vmware-imagebuilder vmware-network-coredump vpxd

---

I'll see if I can get anything out of the VPXD log, as that appears to be our culprit.

Thanks.

MaydayIT · ‎04-17-2019

I checked the vpxd-svcs.log file and found a lot of the following:

2019-04-17T07:39:20.782-06:00 [pool-18-thread-1 WARN com.vmware.cis.server.util.VpxdClient opId=] Cannot handle exception during retry: com.vmware.vim.vmomi.client.exception.ConnectionException: org.apache.http.conn.HttpHostConnectException: Connect to localhost:8085 [localhost/127.0.0.1, localhost/0:0:0:0:0:0:0:1] failed: Connection refused: connect

2019-04-17T07:39:20.782-06:00 [pool-18-thread-1 WARN com.vmware.cis.server.util.impl.KeepAliveTask opId=] Encountered error while trying to keepSessionAlive for connection 7

java.lang.NullPointerException

at com.vmware.vim.vmomi.client.http.impl.HttpProtocolBindingBase.setSession(HttpProtocolBindingBase.java:132)

at com.vmware.cis.server.util.VpxdClient.keepAlive(VpxdClient.java:312)

at com.vmware.cis.server.util.impl.KeepAliveTask.run(KeepAliveTask.java:32)

at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)

at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)

at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)

at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)

at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)

at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)

at java.lang.Thread.run(Thread.java:748)

I've uploaded the entire log in hopes it'll help.

Thanks.

Derek

DylanEps · ‎04-18-2019

Hello, hope this KB is usefull: VMware Knowledge Base

MaydayIT · ‎04-18-2019

This article relates directly to systems that have been sysprepped after VCenter installation. While the system was built from a sysprepped template, the install was started after the machine had already been generalized.

Thanks for the suggestion though.

Derek

msripada · ‎04-18-2019

Hey,

It's failing to connect localhost and at port 8085 which is used by vpxd-svcs.. Validate the log timestamp after service restart using service-control --start vpxd-svcs

If the log is updating with same error, use below command and identify if 8085 used by any other service on windows

Netstat -anob |findstr 8085

Check task manager process pid if there is anything existing in that with the output pid

If nothing returns. .Ping localhost, see it returns ipv6 and yes then disable ipv6 from registry and try rebooting vcentre. .

Interested to know why vpxd-svcs is in started state as it is failing to listen at 8085 which ideally it suppose to do and I believe vpxd failed to connect 8085 as it is dependent on Vpxd-svcs

Thanks

MaydayIT · ‎04-23-2019

Thanks msripada.

I've attached the latest vpxd log file. It seems that it starts then stops after a few minutes. I checked Netstat and it showed nothing else on 8085. I've attached the log file starting a little before the service was restarted and ending a short time later.

I restarted the service again, this time being more careful to note the time. Here's a shorter version.

2019-04-23T14:38:04.319-06:00 [pool-9-thread-7 WARN com.vmware.cis.core.tagging.internal.helper.TaggingAuthzCalculator opId=sps-DTCPoller-605033-168] Unable to fetch VC root id and set it as null

2019-04-23T14:38:24.320-06:00 [pool-9-thread-1 WARN com.vmware.cis.server.util.VpxdClient opId=sps-DTCPoller-605033-168] Cannot handle exception during retry: com.vmware.vim.vmomi.client.exception.ConnectionException: org.apache.http.conn.HttpHostConnectException: Connect to localhost:8085 [localhost/127.0.0.1, localhost/0:0:0:0:0:0:0:1] failed: Connection refused: connect

2019-04-23T14:38:24.320-06:00 [pool-9-thread-1 WARN com.vmware.cis.core.authz.accesscontrol.impl.CheckPrivilegesRouterRiseImpl opId=sps-DTCPoller-605033-168] Error occurred querying the VC Root object URI

com.vmware.cis.server.util.exception.VpxdClientException: java.util.concurrent.ExecutionException: com.vmware.vim.vmomi.client.exception.ConnectionException: org.apache.http.conn.HttpHostConnectException: Connect to localhost:8085 [localhost/127.0.0.1, localhost/0:0:0:0:0:0:0:1] failed: Connection refused: connect

at com.vmware.cis.server.util.VpxdClient.getRootFolder(VpxdClient.java:268)

at com.vmware.cis.server.util.AuthzManagerWrapper.getRootFolder(AuthzManagerWrapper.java:1047)

at com.vmware.cis.core.authz.accesscontrol.impl.CheckPrivilegesRouterRiseImpl.getVcRootObjUriCore(CheckPrivilegesRouterRiseImpl.java:928)

at com.vmware.cis.core.authz.accesscontrol.impl.CheckPrivilegesRouterRiseImpl.getVcRootObjectUri(CheckPrivilegesRouterRiseImpl.java:912)

at com.vmware.cis.core.tagging.internal.helper.TaggingAuthzCalculator.hasPrivilegeOnVcRoot(TaggingAuthzCalculator.java:391)

at com.vmware.cis.core.tagging.internal.impl.TagAssociationManagerImpl.listAllTagAssociationsInfoForObjectType(TagAssociationManagerImpl.java:1354)

at com.vmware.cis.core.tagging.vmodl.MoTagManager.getModifiedTagAssociations(MoTagManager.java:655)

at sun.reflect.GeneratedMethodAccessor109.invoke(Unknown Source)

at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

at java.lang.reflect.Method.invoke(Method.java:498)

at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:333)

at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:190)

at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:157)

at org.springframework.aop.interceptor.JamonPerformanceMonitorInterceptor.invokeUnderTrace(JamonPerformanceMonitorInterceptor.java:115)

at org.springframework.aop.interceptor.AbstractTraceInterceptor.invoke(AbstractTraceInterceptor.java:129)

at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)

at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:213)

at com.sun.proxy.$Proxy93.getModifiedTagAssociations(Unknown Source)

at sun.reflect.GeneratedMethodAccessor109.invoke(Unknown Source)

at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

at java.lang.reflect.Method.invoke(Method.java:498)

at com.vmware.vim.vmomi.server.impl.InvocationTask.run(InvocationTask.java:65)

at com.vmware.vim.vmomi.server.common.impl.RunnableWrapper$1.run(RunnableWrapper.java:47)

at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)

at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)

at java.lang.Thread.run(Thread.java:748)

Caused by: java.util.concurrent.ExecutionException: com.vmware.vim.vmomi.client.exception.ConnectionException: org.apache.http.conn.HttpHostConnectException: Connect to localhost:8085 [localhost/127.0.0.1, localhost/0:0:0:0:0:0:0:1] failed: Connection refused: connect

at com.vmware.vim.vmomi.core.impl.BlockingFuture.get(BlockingFuture.java:81)

at com.vmware.cis.server.util.VpxdClient.getVcServiceInstanceContent(VpxdClient.java:247)

at com.vmware.cis.server.util.VpxdClient.getRootFolder(VpxdClient.java:263)

... 25 more

Caused by: com.vmware.vim.vmomi.client.exception.ConnectionException: org.apache.http.conn.HttpHostConnectException: Connect to localhost:8085 [localhost/127.0.0.1, localhost/0:0:0:0:0:0:0:1] failed: Connection refused: connect

at com.vmware.vim.vmomi.client.common.impl.ResponseImpl.setError(ResponseImpl.java:256)

at com.vmware.vim.vmomi.client.http.impl.HttpExchange.run(HttpExchange.java:51)

at com.vmware.vim.vmomi.client.http.impl.HttpProtocolBindingBase.executeRunnable(HttpProtocolBindingBase.java:226)

at com.vmware.vim.vmomi.client.http.impl.HttpProtocolBindingImpl.send(HttpProtocolBindingImpl.java:110)

at com.vmware.vim.vmomi.client.common.impl.MethodInvocationHandlerImpl$CallExecutor.sendCall(MethodInvocationHandlerImpl.java:613)

at com.vmware.vim.vmomi.client.common.impl.MethodInvocationHandlerImpl$CallExecutor.executeCall(MethodInvocationHandlerImpl.java:594)

at com.vmware.vim.vmomi.client.common.impl.MethodInvocationHandlerImpl.completeCall(MethodInvocationHandlerImpl.java:345)

at com.vmware.vim.vmomi.client.common.impl.MethodInvocationHandlerImpl.invokeFetch(MethodInvocationHandlerImpl.java:329)

at com.vmware.vim.vmomi.client.common.impl.MethodInvocationHandlerImpl.invoke(MethodInvocationHandlerImpl.java:175)

at com.sun.proxy.$Proxy100.getContent(Unknown Source)

at com.vmware.cis.server.util.VpxdClient.getVcServiceInstanceContent(VpxdClient.java:246)

... 26 more

Caused by: org.apache.http.conn.HttpHostConnectException: Connect to localhost:8085 [localhost/127.0.0.1, localhost/0:0:0:0:0:0:0:1] failed: Connection refused: connect

at org.apache.http.impl.conn.HttpClientConnectionOperator.connect(HttpClientConnectionOperator.java:140)

at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:314)

at org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:363)

at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:219)

at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:195)

at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:86)

at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:108)

at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:186)

at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:82)

at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:57)

at com.vmware.vim.vmomi.client.http.impl.HttpExchange.run(HttpExchange.java:45)

... 35 more

Caused by: java.net.ConnectException: Connection refused: connect

at java.net.DualStackPlainSocketImpl.connect0(Native Method)

at java.net.DualStackPlainSocketImpl.socketConnect(DualStackPlainSocketImpl.java:79)

at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)

at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)

at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)

at java.net.PlainSocketImpl.connect(PlainSocketImpl.java:172)

at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)

at java.net.Socket.connect(Socket.java:589)

at org.apache.http.conn.socket.PlainConnectionSocketFactory.connectSocket(PlainConnectionSocketFactory.java:72)

at org.apache.http.impl.conn.HttpClientConnectionOperator.connect(HttpClientConnectionOperator.java:117)

... 45 more

2019-04-23T14:38:24.321-06:00 [pool-9-thread-1 WARN com.vmware.cis.core.tagging.internal.helper.TaggingAuthzCalculator opId=sps-DTCPoller-605033-168] Unable to fetch VC root id and set it as null

2019-04-23T14:38:44.318-06:00 [pool-9-thread-4 WARN com.vmware.cis.server.util.VpxdClient opId=sps-DTCPoller-605033-168] Cannot handle exception during retry: com.vmware.vim.vmomi.client.exception.ConnectionException: org.apache.http.conn.HttpHostConnectException: Connect to localhost:8085 [localhost/127.0.0.1, localhost/0:0:0:0:0:0:0:1] failed: Connection refused: connect

2019-04-23T14:38:44.318-06:00 [pool-9-thread-4 WARN com.vmware.cis.core.authz.accesscontrol.impl.CheckPrivilegesRouterRiseImpl opId=sps-DTCPoller-605033-168] Error occurred querying the VC Root object URI