k get pv,pvc
NAME CAPACITY ACCESS MODES RECLAIM POLICY STATUS CLAIM STORAGECLASS REASON AGE
persistentvolume/pvc-122a109a-0ced-4601-b271-09a0ca43d18f 9Gi RWO Delete Bound default/myclaim2 dt-vsan 71m
persistentvolume/pvc-8b1b8cc0-9ded-49a2-9217-6d7f90a01fd6 9Gi RWO Delete Bound default/myclaim3 dt-vmfs 60m
persistentvolume/pvc-e78fb5fa-bdad-4245-a7e0-954c98d89990 12Gi RWO Delete Bound default/myclaim5 dt-vmfs 19m
NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE
persistentvolumeclaim/myclaim2 Bound pvc-122a109a-0ced-4601-b271-09a0ca43d18f 9Gi RWO dt-vsan 71m
persistentvolumeclaim/myclaim3 Bound pvc-8b1b8cc0-9ded-49a2-9217-6d7f90a01fd6 9Gi RWO dt-vmfs 60m
persistentvolumeclaim/myclaim5 Bound pvc-e78fb5fa-bdad-4245-a7e0-954c98d89990 12Gi RWO dt-vmfs 19m
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled <unknown> Successfully assigned default/nginx3 to tkgs-cluster-2-workers-ksm4n-c599d66cf-j5g8t
Warning FailedAttachVolume 16m (x9 over 18m) attachdetach-controller AttachVolume.Attach failed for volume "pvc-e78fb5fa-bdad-4245-a7e0-954c98d89990" : rpc error: code = Internal desc = observed Error: "ServerFaultCode: NotAuthenticated" is set on the volume "f5dc08e7-520d-48b7-8278-c426bd3f2f9b-e78fb5fa-bdad-4245-a7e0-954c98d89990" on virtualmachine "tkgs-cluster-2-workers-ksm4n-c599d66cf-j5g8t"
Warning FailedMount 47s (x8 over 16m) kubelet, tkgs-cluster-2-workers-ksm4n-c599d66cf-j5g8t Unable to attach or mount volumes: unmounted volumes=[myclaim5], unattached volumes=[myclaim5 default-token-hns22]: timed out waiting for the condition
Warning FailedAttachVolume 17s (x8 over 14m) attachdetach-controller AttachVolume.Attach failed for volume "pvc-e78fb5fa-bdad-4245-a7e0-954c98d89990" : rpc error: code = Internal desc = Watch on virtualmachine "tkgs-cluster-2-workers-ksm4n-c599d66cf-j5g8t" timed out
Spec:
Attacher: csi.vsphere.vmware.com
Node Name: tkgs-cluster-2-workers-ksm4n-c599d66cf-j5g8t
Source:
Persistent Volume Name: pvc-e78fb5fa-bdad-4245-a7e0-954c98d89990
Status:
Attach Error:
Message: rpc error: code = Internal desc = Watch on virtualmachine "tkgs-cluster-2-workers-ksm4n-c599d66cf-j5g8t" timed out
Time: 2021-12-14T09:19:00Z
Attached: false
Anyone faced this?
This issue was fixed after regenerating the workload storage user password. Workload storage user that gets created as part of Supervisor cluster enablement, it's credentials was not working somehow, so regenerated the password and then the issue was resolved.
Hey Dinesh, what do you mean by "regenerating the workload storage user password"?!
Login to one of the SupervisorControlPlaneVMs and run:
kubectl rollout restart deployment vsphere-csi-controller -n vmware-system-csi
this will regenerate Workload Storage User password which is stored in vsphere-config-secret in the same namespace.
This resolved my issue with PVC not bounding to PV, which was failing to be created due to Not Authenticated error.
In my case I was creating Tanzu cluster with additional storage for containerd.