After ESXi host client upgrade from 7 to 8 - "no helthy upstream"/"no certificate or crl found"

[athompson88]

I am attempting to upgrade attempt of my home ESXi server, which is standalone using the free license, and not tied to a vcenter. I first encountered the precheck error, "SHA-1 signature found in host certificate rui.crt with subject". I got past this by trying to generate a new SSL cert (rui.crt, rui.key) and placing it in /etc/vmware/ssl. I thought this was successful as the new cert showed up by itself under Management -> Users And Security -> Certificates. I was able to reboot the host and bring up all the vms without issue. I then proceeded with the upgrade using esxcli, and this also returned "successful."

Upon reboot though, I could SSH into the host, but the web ui wouldn't start, giving me the error "no healthy upstream". After some poking around, I discovered that host processes were not running. I ran "/etc/init.d/hostd start", but it continued to crash. I found the following at the end of "/var/log/hostd.log"

2023-10-26T04:03:06.966Z Er(163) Hostd[535046]: [Originator@6876 sub=Solo] Failed to create SSL context: N7Vmacore3Ssl12SSLExceptionE(SSL Exception: error:05800088:x509 certificate routines::no certificate or crl found)
2023-10-26T04:03:06.967Z Er(163) Hostd[535046]: --> [context]zKq7AVICAgAAAP////8QaG9zdGQAAANZQWxpYnZtYWNvcmUuc28AALXJKAAszSgAqyIfAHAVNQD6GTUAaUg1ASvFimhvc3RkAAHA1YoATV9CAW1rXQHyT10BiXhdARoXUgIdNAJsaWJjLnNvLjYAATXYUw==[/context]

Here are the contents of the SSL directory

[root@hawk:/etc/vmware/ssl] ls -lisa
total 44
18216 4 drwxr-xr-x 1 root root 512 Oct 26 04:02 .
18521 0 -r--r--r-T 1 root root 0 Sep 4 22:01 .#iofiltervp_castore.pem
18287 0 -r--r--r-T 1 root root 0 Sep 4 22:01 .#rui.crt
18292 0 -r-------T 1 root root 0 Sep 4 22:01 .#rui.key
18291 0 -r--r--r-T 1 root root 0 Sep 4 22:01 .#vasa.crt
18290 0 -r-------T 1 root root 0 Sep 4 22:01 .#vasa.key
18120 4 drwxr-xr-x 1 root root 512 Oct 26 04:03 ..
18289 0 -rw-r--r-T 1 root root 0 Sep 4 22:01 castore.pem
18910 4 -rw-r--r-- 1 root root 351 Oct 26 03:30 fipsmodule.cnf
18992 4 -rw-r--r-- 1 root root 3068 Oct 26 03:31 iofiltervp.pem
18922 0 -rw-r--r-- 1 root root 0 Oct 26 03:04 iofiltervp_castore.pem
18288 4 -r--r--r-- 1 root root 1279 Sep 4 22:01 openssl.cnf
19072 4 -rw-r--r-- 1 root root 1363 Oct 26 04:02 rui.crt
19083 4 -rw-r--r-- 1 root root 1135 Oct 26 04:01 rui.csr
18990 4 -rw------- 1 root root 1704 Oct 26 04:01 rui.key
18920 4 -rw-r--r-- 1 root root 1761 Oct 26 03:05 vasa.crt
18919 4 -r-------- 1 root root 1704 Oct 26 03:05 vasa.key
18864 0 -rw-r--r-T 1 root root 0 Sep 4 22:03 vsan_kms_castore.pem
18866 0 -rw-r--r-T 1 root root 0 Sep 4 22:03 vsan_kms_castore_old.pem
18867 0 -rw-r--r-T 1 root root 0 Sep 4 22:03 vsan_kms_client.crt
18865 0 -r-------T 1 root root 0 Sep 4 22:03 vsan_kms_client.key
18868 0 -rw-r--r-T 1 root root 0 Sep 4 22:03 vsan_kms_client_old.crt
18870 0 -r-------T 1 root root 0 Sep 4 22:03 vsan_kms_client_old.key
18869 0 -rw-r--r-T 1 root root 0 Sep 4 22:03 vsanvp_castore.pem

Perhaps I was not able to generate the SSL certificate correctly? I tried regenerating it again, but same result. I figured I should come here and ask experts as I'm a lightweight when it comes to certificates. Here are the commands I used to create the certs

[root@hawk:/etc/vmware/ssl] openssl genrsa -out /etc/vmware/ssl/rui.key 2048
Generating RSA private key, 2048 bit long modulus
****************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************+++++
**********************************************************************************************************************************************************************************************************************************************************************************************************************************************************+++++
e is 65537 (0x10001)
[root@hawk:/etc/vmware/ssl] openssl req -new -nodes -out /etc/vmware/ssl/rui.csr -keyout /etc/vmware/ssl/rui.key -config /etc/vmware/ssl/bak/webclient.cnf
Generating a RSA private key
**********************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************+++++
************************+++++
writing new private key to '/etc/vmware/ssl/rui.key'
-----
[root@hawk:/etc/vmware/ssl] openssl x509 -req -days 4380 -in /etc/vmware/ssl/rui.csr -signkey /etc/vmware/ssl/rui.key -out /etc/vmware/ssl/rui.crt -extensions v3_req -extfile /
etc/vmware/ssl/bak/webclient.cnf
Signature ok
subject=/C=US/ST=OH/L=Columbus/O=<my domain name>/OU=Family/CN=hawk.<my domain name>
Getting Private key

[athompson88 · Accepted solution]

Follow-up. I found the following:

https://blogs.vmware.com/cloud-foundation/2020/04/14/replacing-vmware-esxi-ssl-certificate-in-vmware...

I think I missed a step during certificate replacement. Also the error I noted in the original post about failing to create the SSL context is present after I rolled back all changes made prior to changing the SSL and performing the upgrade. 

[athompson88]

I tried again using the directions on the link in my previous post, and this time it seems to have worked. I'm going to repeat it a second time, but unless that fails, I think it was a case of using an incorrect method.

[athompson88]

Confirmed. I was able to perform a second successful upgrade. 

[percodata]

@athompson88- wanted to drop in and say thank you so much for taking the time to come back and update your own thread, confirming what the issue / solution was!

Could I make a suggestion though - if it's possible to edit thread titles here, consider adding "ESXi Host Client" and "No healthy upstream" to it, for easier ID in search engines? The latter is what I spent most of my time searching as it was the most obvious error - and the former to help differentiate from the countless threads about this error which relate to vCenter, rather the the ESXi Host Client.

Anyway - I just faced the same problem upgrading from an older 7.x ESXi version to version 8 Update 2. Upgrade appeared to complete successfully, but when trying to load the ESXi Host client (like you, also just running a single host in a home environment, no vCenter) it simply displayed a brief "No healthy upstream" error message.

I actually noticed the "Failed to create SSL context: N7Vmacore3Ssl12SSLExceptionE(SSL Exception: error:05800088:x509 certificate routines::no certificate or crl found)" in hostd.log quite early in my troubleshooting, but some info I read lead me astray - lead me to think that wasn't the cause of the problem. Messed around with DNS, NTP, and a few other things, and finally returned to trying the steps in the link you shared (https://blogs.vmware.com/cloud-foundation/2020/04/14/replacing-vmware-esxi-ssl-certificate-in-vmware...), which fixed the issue.

I'm going to leave a couple of additional notes here, for anyone else who searches for / stumbles across this post in the future:

1) When following https://blogs.vmware.com/cloud-foundation/2020/04/14/replacing-vmware-esxi-ssl-certificate-in-vmware... you need to complete the steps under "Create a self-signed certificate using OpenSSL" and "Steps to replace certificate for ESXi host" - works fine while skipping the vCenter-related sections in the middle.

2) You can complete "Create a self-signed certificate using OpenSSL" steps on the ESXi host itself. I did run into an error when trying to execute "openssl req -x509 -newkey rsa:4096 -keyout key.pem -out cert.pem -days 3650 -nodes" - didn't capture it unfortunately. But the solution was to create a default-value openssl.cfg file (I grabbed the contents from the latest openssl 3.0.x release tarball), then add "-config openssl.cfg" to the aforementioned command.

3) Change "-days 365", in each command it's referenced, to "-day 3650", so your new certificate lasts for 10 years instead of one.

[athompson88]

Yes, I did hit the same error you did, and came to the same conclusion. Since you brought it up, let me post the generic self-signed certificate cfg template I used to overcome that issue. This would be used as part of step 1 in the "Replacing VMware ESXi SSL Certificate in VMware Cloud Foundation" document which is what I ultimately used (again, leaving out the middle vCenter related steps).

[req]
default_bits = 4096
default_keyfile = rui.key
distinguished_name = req_distinguished_name
encrypt_key = no
prompt = no
string_mask = nombstr
req_extensions = v3_req

[v3_req]
basicConstraints = CA:FALSE
keyUsage = digitalSignature, keyEncipherment, dataEncipherment
extendedKeyUsage = serverAuth, clientAuth
subjectAltName = DNS:<YOUR FQDN HERE>

[req_distinguished_name]
countryName = <YOUR 2CHAR COUNTRY CODE>
stateOrProvinceName = <YOUR 2 CHAR STATE/PROVINCE>
localityName = <YOUR CITY>
0.organizationName = <WHATEVER>
organizationalUnitName = <WHATEVER>
commonName = <YOUR FQDN>

The command to use becomes

openssl req -x509 -newkey rsa:4096 -keyout key.pem -out cert.pem -days 3650 -nodes --cert <path_to_self_signed_cert_template>