Re: PEM format issue with upgrade coordinator

mthree18 · ‎08-17-2023

Referencing this article: https://kb.vmware.com/s/article/93518

I had replaced my certificates in NSX with PEM format certs. All works well, except when upgrading from 3.2 to 4.1.1, the upgrade coordinator cannot migrate the cluster from the 3.2 node to the 4.1.1 node to finalize the 3rd nodes upgrade. The error reported is due to the certificate PEM format as seen in this document.

1. Is there a way to modify the certificate on the NSX manager outside of uploading the certificate with the extraneous data redacted? As in, I want to manually go edit the certificate on the nodes.

2. If I cannot manually edit eh certificate on the nodes, can I upload the extraneous data redacted certificate? Will it utilized the same UUID give to the original or treat it as a new certificate? If its new, that does not help me as I would then need to modify the certs on all the nodes while I am in precarious state as its currently still mid upgrade.

RobinHarmsen · ‎10-10-2023

@mthree18 We seem to have hit this exact same issue while upgrade one of our NSX-T 4.0 environments into 4.1.1

Unfortunalty the KB does not state how to fix this.....
So I am really wondering whether you managed to fix this issue and how.

mthree18 · ‎10-10-2023

Unfortunately, what happened is that I grew so frustrated having replaced all of the certificates manually only to find that the additional attributes in the PEM format muddied the water. After finding this and failing the upgrade, I changed my course of action. You can use the Multi-NSX configuration with vCenter. When I updated all my NSX infra to support multiple NSX, I deployed an entire new 4.1 solution and registered it as well to the same vCenter. Then, I went through the process of removing NSX from the clusters from the old NSX and re-preparing them with the new NSX. It was a painful process, but saved much more time than trying to fix the certificate issue.

I am not sure that is viable for you, but that is what I did.

RobinHarmsen · ‎10-11-2023

Well I am not willing to replace the whole NSX infra/setup...

But we manged (with help from VMware support) to fix this.
Altough we had hit a slightly different KB it still had to do with the certicates..

We ended up by fixing the certificate on the managers by hand.. specifically /home/secureall/secureall/.store/.tomcat_cert.pem
This solved the issue with the two broken nodes so we could upgrade the last one.

The last one hit a different issue and fixing by hand was not possible..
That one I fixed by temporaraly assinging a self segned certificate and aftetr that replacing it with the correct and fixed certificate.

mthree18 · ‎10-11-2023

As I said, I was not sure you would be willing/able to do the Multi-NSX support migration and build out new. I was on a short timeline and didn't have the ability to delay any further and support was not able to address the issue quickly. I was looking for the manual process to modify the certificates as well, but unfortunately, support never was able to detail it to me. I am glad you found an easier resolution. You do see now in the updated cert manager, it has the details of where it is in use now, whereas prior, we had to use the API to determine a great deal of that data... Big improvement.