VCSA with Embedded PSC v6.7 (Build 9451876)
VMCA configured as Subordinate CA to a Windows 2012 R2 Enterprise Root CA. (SHA256 Hash / 2048 bit Key)
VMCA replaces the SSL certificate on a ESXi v6.5 (Build 5969303) host and the 'certification path' is complete. All works as expected, no browser errors.
VMCA replaces the SSL certificate on a ESXi v6.7 (Build 8169922) host and the 'certification path' is incomplete. Still get the standard browser errors. The root CA and VMCA certificates are NOT in the path, only the ESXi host certificate!
ESXi v6.5 Host - Complete Certification Path. |
---|
A dump of the SSL connection using the TestSSLServer utility (GitHub - pornin/TestSSLServer ) shown below. Connection: mc-esxi-v-204.momusconsulting.com:443 SNI: mc-esxi-v-204.momusconsulting.com TLSv1.0: server selection: uses client preferences 3-- (key: RSA) RSA_WITH_AES_128_CBC_SHA 3-- (key: RSA) RSA_WITH_AES_256_CBC_SHA 3f- (key: RSA) ECDHE_RSA_WITH_AES_128_CBC_SHA 3f- (key: RSA) ECDHE_RSA_WITH_AES_256_CBC_SHA TLSv1.1: idem TLSv1.2: server selection: enforce server preferences 3f- (key: RSA) ECDHE_RSA_WITH_AES_256_GCM_SHA384 3f- (key: RSA) ECDHE_RSA_WITH_AES_128_GCM_SHA256 3-- (key: RSA) RSA_WITH_AES_256_GCM_SHA384 3-- (key: RSA) RSA_WITH_AES_128_GCM_SHA256 3f- (key: RSA) ECDHE_RSA_WITH_AES_256_CBC_SHA384 3f- (key: RSA) ECDHE_RSA_WITH_AES_256_CBC_SHA 3f- (key: RSA) ECDHE_RSA_WITH_AES_128_CBC_SHA256 3f- (key: RSA) ECDHE_RSA_WITH_AES_128_CBC_SHA 3-- (key: RSA) RSA_WITH_AES_256_CBC_SHA256 3-- (key: RSA) RSA_WITH_AES_256_CBC_SHA 3-- (key: RSA) RSA_WITH_AES_128_CBC_SHA256 3-- (key: RSA) RSA_WITH_AES_128_CBC_SHA ========================================= +++++ SSLv3/TLS: 1 certificate chain(s) +++ chain: length=3 names match: yes includes root: yes signature hash(es): SHA-256 + certificate order: 0 thumprint: A18830247B90395EE003D706CE3AEB3CDA96BC6D serial: E032A1675443F48D subject: EMAILADDRESS=admin@momusconsulting.com,CN=mc-esxi-v-204.momusconsulting.com,OU=Momus Labs,O=Momus Consulting,L=Basingstoke,ST=Basingstoke,C=GB issuer: CN=VMCA-mc-vcsa-v-204,OU=Momus Labs,O=Momus Consulting,L=Basingstoke,ST=Hampshire,C=GB valid from: 2018-10-06 14:22:12 UTC valid to: 2020-10-05 12:06:47 UTC key type: RSA key size: 2048 sign hash: SHA-256 server names: mc-esxi-v-204.momusconsulting.com + certificate order: 1 thumprint: 6313EF9061D1ED748298F0DB7D693F6CC2099046 serial: 5D0000000BA3C47E6295F579B400000000000B subject: CN=VMCA-mc-vcsa-v-204,OU=Momus Labs,O=Momus Consulting,L=Basingstoke,ST=Hampshire,C=GB issuer: CN=Momus Root CA on mc-addc-v-101,DC=momusconsulting,DC=com valid from: 2018-10-06 12:06:47 UTC valid to: 2020-10-05 12:06:47 UTC key type: RSA key size: 2048 sign hash: SHA-256 + certificate order: 2 thumprint: A3BD98D6B6C712A510E11669A84D0571C2D2F0F1 serial: 65F1DEEF09DD1A9A436075662D731F0F subject: CN=Momus Root CA on mc-addc-v-101,DC=momusconsulting,DC=com issuer: CN=Momus Root CA on mc-addc-v-101,DC=momusconsulting,DC=com valid from: 2018-10-05 15:11:29 UTC valid to: 2028-10-05 15:21:28 UTC key type: RSA key size: 2048 sign hash: SHA-256 (self-issued) ========================================= Server compression support: no Server sends a random system time. Secure renegotiation support: yes Encrypt-then-MAC support (RFC 7366): no SSLv2 ClientHello format (for SSLv3+): yes Minimum EC size (no extension): 256 Minimum EC size (with extension): 256 ECDH parameter reuse: no Supported curves (size and name) ('*' = selected by server): * 256 secp256r1 (P-256) ========================================= WARN[CS006]: Server supports cipher suites with no forward secrecy. |
ESXi v6.7 Host - Incomplete Certification Path. |
---|
Again, a dump of the SSL connection is shown below. Connection: mc-esxi-v-205.momusconsulting.com:443 SNI: mc-esxi-v-205.momusconsulting.com TLSv1.2: server selection: enforce server preferences 3f- (key: RSA) ECDHE_RSA_WITH_AES_256_GCM_SHA384 3f- (key: RSA) ECDHE_RSA_WITH_AES_128_GCM_SHA256 3-- (key: RSA) RSA_WITH_AES_256_GCM_SHA384 3-- (key: RSA) RSA_WITH_AES_128_GCM_SHA256 3f- (key: RSA) ECDHE_RSA_WITH_AES_256_CBC_SHA384 3f- (key: RSA) ECDHE_RSA_WITH_AES_256_CBC_SHA 3f- (key: RSA) ECDHE_RSA_WITH_AES_128_CBC_SHA256 3f- (key: RSA) ECDHE_RSA_WITH_AES_128_CBC_SHA 3-- (key: RSA) RSA_WITH_AES_256_CBC_SHA256 3-- (key: RSA) RSA_WITH_AES_256_CBC_SHA 3-- (key: RSA) RSA_WITH_AES_128_CBC_SHA256 3-- (key: RSA) RSA_WITH_AES_128_CBC_SHA ========================================= +++++ SSLv3/TLS: 1 certificate chain(s) +++ chain: length=1 names match: yes includes root: no signature hash(es): SHA-256 + certificate order: 0 thumprint: 9CB7BEC3BD58491A36069B182093F22BE9813042 serial: FD682ECC9662D00C subject: EMAILADDRESS=admin@momusconsulting.com,CN=mc-esxi-v-205.momusconsulting.com,OU=Momus Labs,O=Momus Consulting,L=Basingstoke,ST=Basingstoke,C=GB issuer: CN=VMCA-mc-vcsa-v-204,OU=Momus Labs,O=Momus Consulting,L=Basingstoke,ST=Hampshire,C=GB valid from: 2018-10-06 14:44:04 UTC valid to: 2020-10-05 12:06:47 UTC key type: RSA key size: 2048 sign hash: SHA-256 server names: mc-esxi-v-205.momusconsulting.com ========================================= Server compression support: no Server sends a random system time. Secure renegotiation support: yes Encrypt-then-MAC support (RFC 7366): no SSLv2 ClientHello format (for SSLv3+): yes Minimum EC size (no extension): 256 Minimum EC size (with extension): 256 ECDH parameter reuse: no Supported curves (size and name) ('*' = selected by server): * 256 secp256r1 (P-256) ========================================= WARN[CS006]: Server supports cipher suites with no forward secrecy. |
Any ideas?
Thanks
M
Found out how to fix it; but I cannot take the glory..! That goes to to @weg0t0eleven on Reddit (https://www.reddit.com/r/vmware/comments/8z4zal/certificate_chain_on_esxi_nodes/)
On each ESXi 6.7 / 6.7 U1 host, edit the /etc/vmware/rhttpproxy/config.xml file to remove the <!-- and --> comment tags for the <keyStoreFile> line (approx line 77).
Save the file and reboot the host.
/etc/vmware/rhttpproxy/config.xml |
---|
<!-- Remove the following node to disable SSL --> <ssl> <!-- The server private key file --> <privateKey>/etc/vmware/ssl/rui.key</privateKey>
<!-- The server side certificate file --> <certificate>/etc/vmware/ssl/rui.crt</certificate>
<!-- Client-side CAFile verify location --> <!-- <keyStoreFile>/etc/vmware/ssl/castore.pem</keyStoreFile> --> </ssl> |
<!-- Remove the following node to disable SSL --> <ssl> <!-- The server private key file --> <privateKey>/etc/vmware/ssl/rui.key</privateKey>
<!-- The server side certificate file --> <certificate>/etc/vmware/ssl/rui.crt</certificate>
<!-- Client-side CAFile verify location --> <keyStoreFile>/etc/vmware/ssl/castore.pem</keyStoreFile> </ssl> |
Once you have made this change on a 6.7 host you can upgrade to 6.7 U1 and the certificate chain remains complete.
If you upgrade a 6.5 host to 6.7/6.7U1 you will need to do make this change & reboot after the upgrade has completed.
Happy days.
I've come across the same issue. I updated my ESXi 6.7 host to 6.7.0, 10302608 and renewed the certificate and am still seeing the same behavior.
gorciakj
It's good to know that I am not alone! :smileygrin:
What is your vCenter version/build/platform?
I haven't had a chance to test it with a 6.7 U1 VCSA and 6.7 U1 ESXi Host.
M
Also running VCSA 6.7 (Build 9433931)
VMCA also configured as a subordinate to Server 2012 R2 CA
The ESXi 6.7 host I updated was to 6.7U1, have not tried updating my VCSA yet.
All my 6.5 hosts have a complete certificate chain just as you've mentioned as well.
Thanks again gorciakj
I will try upgrading my test system at the weekend to see if a 6.7U1 VCSA makes any difference!
I will post what I find up on this thread.
M
gorciakj I found some time today to test this with 6.7 U1 VCSA. Sadly the issue is still there for the 6.7 U1 ESXi hosts!
VCSA with Embedded PSC v6.7 U1 (Build 10244745) - Fresh install NOT an upgrade.
VMCA configured as Subordinate CA to a Windows 2012 R2 Enterprise Root CA. (SHA256 Hash / 2048 bit Key)
:smileycheck: VMCA replaces the SSL certificate on a ESXi v6.5 U1 (Build 5969303) host and the 'certification path' is complete. All works as expected, no browser errors.
:smileycheck: VMCA replaces the SSL certificate on a ESXi v6.5 U2 (Build 8294253) host and the 'certification path' is complete. All works as expected, no browser errors.
:smileyx: VMCA replaces the SSL certificate on a ESXi v6.7 U1 (Build 10302608) host and the 'certification path' is incomplete. Still get the standard browser errors. The root CA and VMCA certificates are NOT in the path, only the ESXi host certificate!
I don't have any support, so I cannot raise a SR to take this any further.
M
Found out how to fix it; but I cannot take the glory..! That goes to to @weg0t0eleven on Reddit (https://www.reddit.com/r/vmware/comments/8z4zal/certificate_chain_on_esxi_nodes/)
On each ESXi 6.7 / 6.7 U1 host, edit the /etc/vmware/rhttpproxy/config.xml file to remove the <!-- and --> comment tags for the <keyStoreFile> line (approx line 77).
Save the file and reboot the host.
/etc/vmware/rhttpproxy/config.xml |
---|
<!-- Remove the following node to disable SSL --> <ssl> <!-- The server private key file --> <privateKey>/etc/vmware/ssl/rui.key</privateKey>
<!-- The server side certificate file --> <certificate>/etc/vmware/ssl/rui.crt</certificate>
<!-- Client-side CAFile verify location --> <!-- <keyStoreFile>/etc/vmware/ssl/castore.pem</keyStoreFile> --> </ssl> |
<!-- Remove the following node to disable SSL --> <ssl> <!-- The server private key file --> <privateKey>/etc/vmware/ssl/rui.key</privateKey>
<!-- The server side certificate file --> <certificate>/etc/vmware/ssl/rui.crt</certificate>
<!-- Client-side CAFile verify location --> <keyStoreFile>/etc/vmware/ssl/castore.pem</keyStoreFile> </ssl> |
Once you have made this change on a 6.7 host you can upgrade to 6.7 U1 and the certificate chain remains complete.
If you upgrade a 6.5 host to 6.7/6.7U1 you will need to do make this change & reboot after the upgrade has completed.
Happy days.
Thanks for the very good explanation how to solve it, I also had to fight with the same problem after I enrolled a new VMCA certificate!
I finally fixed it with a find and replace "sed" command and "cssh" in paralell on many affected hosts in one step:
cd /etc/vmware/rhttpproxy/
grep -q "<version>6.6.0.0</version>" config.xml && sed -i.backup "/Client-side\ CAFile\ verify\ location/ {n;s/<\!\-\- \(.*\) \-\->/\1/}" config.xml
/etc/init.d/rhttpproxy restart
Sed creates a backup(config.xml.backup) of the original file in the same directory.
VMware posted this KB entry regarding the same issue:
wagnewal nice, very nice..! Thank you for sharing!
It's also good to see that VMware now have a KB for this issue; fingers crossed for a fix in an 6.7 U2.
Cheers
M
Hi there,
Thank you so much, worked for me as well (6.7 U1 - latest patches) !
Reading the ESXi v6.7 Update 2 release notes it looks like this issue has been fixed. I haven't tested it myself yet.
VMware ESXi 6.7 Update 2 Release Notes
PR 2212140: Renewing a host certificate might not push the full chain of trust to the ESXi host |
---|
When you renew a certificate, only the first certificate from the supplied chain of trust might be stored on the ESXi host. Any intermediate CA certificates are truncated. Because of the missing certificates, the chain to the root CA cannot be built. This leads to a warning for an untrusted connection. This issue is resolved in this release. |
I have been meaning to update this for a while...
Whatever @VMware fixed with PR2212140 in the ESXi v6.7 U2 release it was not this!
This issue persists with ESXi v6.7 U2 (Build 13006603) and also with the recent release of ESXi v6.7 U3 (Build 14320388).
Editing the /etc/vmware/rhttpproxy/config.xml file as per the posts above still works.
M