Tanzu Mission Control Self-Managed Authentication Issue

[abdurrahman12]

Hello,

I have installed TMC Self-Managed 1.1.0 on my demo cluster which is based on k8s v1.26 to test the features.

I am using Microsoft Active Directory as IDP.

The installation is successfull and all TMC Packages are Succeed.

When I try to login I see the TMC main page. then  I click to login and I see the following error message:

```

Client authentication failed (e.g., unknown client, no client authentication included, or unsupported authentication method). The requested OAuth 2.0 Client does not exist.

```

I have checked the logs of the pinniped supervisor pods, I see following logs:

```
{"level":"error","timestamp":"2024-01-24T18:53:48.184329Z","caller":"/root/go/pkg/mod/k8s.io/apiserver@v0.26.3/pkg/server/dynamiccertificates/tlsconfig.go:275$dynamiccertificates.(*DynamicServingCertificateController).processNextWorkItem","message":"key failed with : not loading an empty serving certificate from \"supervisor-serving-cert\"\n"}
{"level":"error","timestamp":"2024-01-24T18:53:48.189928Z","caller":"/root/go/pkg/mod/k8s.io/apiserver@v0.26.3/pkg/server/dynamiccertificates/tlsconfig.go:275$dynamiccertificates.(*DynamicServingCertificateController).processNextWorkItem","message":"key failed with : not loading an empty serving certificate from \"supervisor-serving-cert\"\n"}
{"level":"error","timestamp":"2024-01-24T18:53:48.200000Z","caller":"/root/go/pkg/mod/k8s.io/apiserver@v0.26.3/pkg/server/dynamiccertificates/tlsconfig.go:275$dynamiccertificates.(*DynamicServingCertificateController).processNextWorkItem","message":"key failed with : not loading an empty serving certificate from \"supervisor-serving-cert\"\n"}
{"level":"error","timestamp":"2024-01-24T18:53:48.220409Z","caller":"/root/go/pkg/mod/k8s.io/apiserver@v0.26.3/pkg/server/dynamiccertificates/tlsconfig.go:275$dynamiccertificates.(*DynamicServingCertificateController).processNextWorkItem","message":"key failed with : not loading an empty serving certificate from \"supervisor-serving-cert\"\n"}
{"level":"error","timestamp":"2024-01-24T18:53:48.260888Z","caller":"/root/go/pkg/mod/k8s.io/apiserver@v0.26.3/pkg/server/dynamiccertificates/tlsconfig.go:275$dynamiccertificates.(*DynamicServingCertificateController).processNextWorkItem","message":"key failed with : not loading an empty serving certificate from \"supervisor-serving-cert\"\n"}
{"level":"info","timestamp":"2024-01-24T18:53:48.283297Z","caller":"/root/go/pkg/mod/k8s.io/client-go@v0.26.3/tools/cache/shared_informer.go:280$cache.WaitForNamedCacheSync","message":"Caches are synced for RequestHeaderAuthRequestController\n"}
{"level":"info","timestamp":"2024-01-24T18:53:48.284663Z","caller":"/root/go/pkg/mod/k8s.io/client-go@v0.26.3/tools/cache/shared_informer.go:280$cache.WaitForNamedCacheSync","message":"Caches are synced for client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file\n"}
{"level":"info","timestamp":"2024-01-24T18:53:48.284737Z","caller":"/root/go/pkg/mod/k8s.io/apiserver@v0.26.3/pkg/util/flowcontrol/apf_controller.go:366$flowcontrol.(*configController).Run","message":"Running API Priority and Fairness config worker\n"}
{"level":"info","timestamp":"2024-01-24T18:53:48.284788Z","caller":"/root/go/pkg/mod/k8s.io/apiserver@v0.26.3/pkg/util/flowcontrol/apf_controller.go:369$flowcontrol.(*configController).Run","message":"Running API Priority and Fairness periodic rebalancing process\n"}
{"level":"info","timestamp":"2024-01-24T18:53:48.284943Z","caller":"/root/go/pkg/mod/k8s.io/client-go@v0.26.3/tools/cache/shared_informer.go:280$cache.WaitForNamedCacheSync","message":"Caches are synced for client-ca::kube-system::extension-apiserver-authentication::client-ca-file\n"}
{"level":"error","timestamp":"2024-01-24T18:53:48.285071Z","caller":"/root/go/pkg/mod/k8s.io/apiserver@v0.26.3/pkg/server/dynamiccertificates/tlsconfig.go:275$dynamiccertificates.(*DynamicServingCertificateController).processNextWorkItem","message":"key failed with : not loading an empty serving certificate from \"supervisor-serving-cert\"\n"}
{"level":"error","timestamp":"2024-01-24T18:53:48.286560Z","caller":"/root/go/pkg/mod/k8s.io/apiserver@v0.26.3/pkg/server/dynamiccertificates/tlsconfig.go:275$dynamiccertificates.(*DynamicServingCertificateController).processNextWorkItem","message":"key failed with : not loading an empty serving certificate from \"supervisor-serving-cert\"\n"}
{"level":"error","timestamp":"2024-01-24T18:53:48.342434Z","caller":"/root/go/pkg/mod/k8s.io/apiserver@v0.26.3/pkg/server/dynamiccertificates/tlsconfig.go:275$dynamiccertificates.(*DynamicServingCertificateController).processNextWorkItem","message":"key failed with : not loading an empty serving certificate from \"supervisor-serving-cert\"\n"}
{"level":"info","timestamp":"2024-01-24T18:53:48.385788Z","caller":"/root/go/pkg/mod/k8s.io/client-go@v0.26.3/tools/leaderelection/leaderelection.go:248$leaderelection.(*LeaderElector).acquire","message":"attempting to acquire leader lease tmc-local/pinniped-supervisor...\n"}
{"level":"info","timestamp":"2024-01-24T18:57:12.857732Z","caller":"/root/go/pkg/mod/k8s.io/client-go@v0.26.3/tools/leaderelection/leaderelection.go:258$leaderelection.(*LeaderElector).acquire.func1","message":"successfully acquired lease tmc-local/pinniped-supervisor\n"}
```

There are some error messages but I don't think that cause the client not found issue.

I have checked auth-manager-server pod logs, found following:

```

time="2024-01-24T14:03:25Z" level=info msg="Getting Provider Metadata: https://pinniped-supervisor.tmc.domain.co/provider/pinniped/.well-known/openid-configuration" idp=oidc-pinniped
2024/01/24 14:03:25 Serving olympus authentication manager at http://[::]:36015
time="2024-01-24T14:03:25Z" level=info msg="loaded 1 session key(s)"
time="2024-01-24T14:03:25Z" level=info msg="Session key loaded: Zhy..."
time="2024-01-24T14:03:25Z" level=info msg="Getting Provider Metadata: https://pinniped-supervisor.tmc.domain.co/provider/pinniped/.well-known/openid-configuration" idp=oidc-pinniped
2024/01/24 14:03:25 Serving olympus authentication manager at https://[::]:8443
time="2024-01-24T18:37:10Z" level=info msg="finished HTTP call with code 307 Temporary Redirect" X-Request-ID=56c18dcd-c8bf-4d9d-bd1e-e4bc75646d31 http.host=auth.tmc.domain.co http.proto_major=2 http.request.length_bytes=0 http.request.method=GET http.request.referer= http.request.user_agent="Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:121.0) Gecko/20100101 Firefox/121.0" http.response.length_bytes=0 http.response.status=307 http.time_ms=0.47 http.url.path=/api/v1/login peer.address=192.0.3.210 peer.port=41030 span.kind=server system=http
time="2024-01-24T18:56:14Z" level=info msg="finished HTTP call with code 307 Temporary Redirect" X-Request-ID=24372986-381f-4f76-b9f4-d24949c8a27e http.host=auth.tmc.domain.co http.proto_major=2 http.request.length_bytes=0 http.request.method=GET http.request.referer= http.request.user_agent="Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:121.0) Gecko/20100101 Firefox/121.0" http.response.length_bytes=0 http.response.status=307 http.time_ms=0.34 http.url.path=/api/v1/login peer.address=192.0.1.242 peer.port=34918 span.kind=server system=http

```

I am wondering what can cause this issue? and how to troubleshoot.

[JohanL1]

Hello,

It sounds you have missmatch with certificate on cert-manager.

verify if tmc using right certificate against cert-manager using same.

kubectl get secret -n cert-manager tmcsm-issuer -o=jsonpath="{.[data.ca](http://data.ca/)\.crt}" | base64 -d

if you have done some redeployed to the cluster to get it working, I would recommend cleaning up the namespace (tmc-local) or deleting it and recreate it