VMware Cloud Community
MC1903
Enthusiast
Enthusiast
Jump to solution

ERROR certificate-manager ERROR:: INVALID_KEY, the private key doesnot match the certificate.

This VCSA has been migrated from a Windows vCenter Server 6.5 Update 1 (build-5973321), and then upgraded to vCenter Appliance 6.7 Update 3t (build-22509723), before finally being upgraded to vCenter Appliance 8.0 Update 2b (build-23319993). I also had is PNID changed after it was upgraded to 8.0U2b.

I am trying Certificate Manager v8.0 option #2 (Replace VMCA Root certificate with Custom Signing Certificate and replace all Certificates) to replace the VMCA certificate with a subordinate CA cert from my enterprise signing CA.

Seconds after starting stage #2. I get the error "ERROR certificate-manager ERROR:: INVALID_KEY, the private key doesnot match the certificate. Please provide a valid certificate and Key pair." and it starts a rollback.

I have done this 100's of times with previous versions of vCenter Server over the years and never seen this error.

I notice the default RSA key size has changed to 3072 from 2048 in this version of Certificate Manager, but what else?

Do I need to amend my Windows Subordinate CA template for version 8.0 vCenter Servers? I cannot see an updated KB for this.

Is anyone aware of Certificate Manager 8.0 issues with vCenter 8.0 Update 2b?

I have tried resetting all certs (option #8), which completes OK; but retrying option #2 fails with same message.

I will do some more digging tonight/tomorrow, but would appreciate any feedback if anyone is aware / has also had this issue.

Cheers

Labels (2)
0 Kudos
1 Solution
2 Replies
MC1903
Enthusiast
Enthusiast
Jump to solution

Something odd is happening with Certificate Manager version 8.0 and/or certool version 2.0.0.

Every CSR generated by them, that is signed by my CA fails to import with the same error.

I tested with a vCenter Appliance 7.0.3 (build-20990077), generated the keys & the CSR using Certificate Manager, signed it with the same CA/Template and it imported & applied without error.

I have worked round my issue on the 8.0 U2b appliance by using OpenSSL to generate the public/private keys, as well as the csr.

*** FOLLOW ON AT YOUR OWN RISK - THIS HAS HAD VERY LIMITED TESTING ***

*** IF YOU HAVE THE SAME/SIMILAR ISSUE AND CAN OPEN AN SR WITH GSS, YOU SHOULD DO SO ***

 

root@mc-vcsa-v-220 [ /tmp ]# openssl genpkey -algorithm RSA -out /tmp/vmca_issued_key.key -pkeyopt rsa_keygen_bits:4096
...+..+...+.+..+....+......+........+......+.+..+.............+..+.+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++*.......+...+........+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++*.+..+....+.........+...........+.+..+.......+...+..+...+......+............+.........+.+.....+.......+......+......+...+...+...+.....+.......+..+............+.+...+.....+.+..................+.................+...+...+....+........+.+..+..................+.........+.....................+.+...+...+.....+....+.....+.+...+.........+.....+............+...+.+.....+.......+...+...........+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
.....+.+........+....+..+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++*..+...............+...+...+......+....+.........+.....+...+....+...+..+.............+...+.....+.........+..........+......+........+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++*........+..............+....+..+.+........+....+..+................+.....+............+...+.......+...+..+...+.......+.....+....................................+.......+...+.....+.........+............+....+............+..+.+..+.......+.....+.+.....+.+.....+.+........+.+.....................+.....+...+......+......+..................+....+............+...+...+..+.............+..+.+.....+.+...+...+.....+...............+...+.....................+...................+.........+..+.........+.........+...+.........+...+.......+...+........+..........+.....+....+........+.+........................+...+.................+.+............+.........+.........+...........+...+...+....+........+.+..+..................+..............................+....+..+.+..................+..+.......+.....+..........+..................+..+...+...+.+.....................+.....+....+.....+.............+........+.......+...............+........+.......+.....+......+.......+.........+..............+....+.....+......+....+...........+...................+........+...+.+...+.....+......+..........+......+........+...+....+...+........+...............+...+....+...+...+......+..................+...........+.+.....+....+...........+.+..+...+....+...+...........+....+..+...............+...+..........+.......................+.............+......+.....+....+.........+.....+...+...........................................+......+......+...+..+...+.........+.+......+...............+......+.....+...+...................+........+.........+....+..+...+.......+.................+.....................+............+...............+...+....+..+..........+........+...+...+....+............+......+...............+.....+...+............+...+.......+...............+..+...............+.......+..................+.....+.+.....+......+....+..+.......+.................+......+...+....+...............+.....+.......+...+...+............+..+................+.........+......+.....+.........+......+.......+..+...+.+.....................+......+.........+...+..+.........+.+........+......+.......+..+......+.+...+................................+.........+.+...+...+.....+....+......+...+......+..............+.....................+.+...+.........+..+...+..........+.....+..................+.............+......+..................+.....................+..+.....................................+.....+.+.....+....+..................+.....+....+..................+..+.+...+.....+......+..........+.........+..+...+.........+.......+......+..+..............................+......+..........+.....+......+......+....+..............+.........+.+......+..............+..........+............+..+......+......+.+........+......+....+..+....+...........+.......+...........+.+...+..+............................+...+............+........+..........+..+.+.................+...+.......+..+............+.+....................+...+......+...............+.+........................+...+.....+......+....+...........+....+.....+......+...............+.+.....+......+...+.+.........+.........+......+......+.....+.........+.........+..........+..+.+.........+...........+...+.......+.........+...........+...+....+.....+............+......+...+....+...........+......+...+............+.+........+....+.....+..................+..........+.........+...+......+..+............+.......+...+..+..........+......+...+.....+.........+...+....+...+.................+.+..+................+............+.........+........+.....................+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
root@mc-vcsa-v-220 [ /tmp ]# openssl rsa -in /tmp/vmca_issued_key.key -pubout -out /tmp/pubkey.pub
writing RSA key
root@mc-vcsa-v-220 [ /tmp ]# cat << EOF > /tmp/openssl.cnf
[ req ]
default_bits		= 4096
encrypt_key			= yes
default_md          = sha256
utf8                = yes
prompt              = no
distinguished_name  = req_distinguished_name
req_extensions      = v3_intermediate_ca

[ req_distinguished_name ]
countryName                 = GB
commonName                  = mc-vcsa-v-220.momusconsulting.com
organizationName            = Momus Consulting
organizationalUnitName      = Momus Labs
stateOrProvinceName         = Hampshire
localityName                = Basingstoke
emailAddress                = certificate-admin@momusconsulting.com

[ v3_intermediate_ca ]
subjectKeyIdentifier        = hash
basicConstraints            = critical, CA:true
keyUsage                    = critical, digitalSignature, cRLSign, keyCertSign
subjectAltName              = IP:10.10.1.220, DNS:mc-vcsa-v-220.momusconsulting.com
EOF
root@mc-vcsa-v-220 [ /tmp ]# 

root@mc-vcsa-v-220 [ /tmp ]# openssl req -new -key /tmp/vmca_issued_key.key -out /tmp/vmca_issued_csr.csr -config /tmp/openssl.cnf
root@mc-vcsa-v-220 [ /tmp ]# 

 

 

Sign the CSR on the Enterprise signing CA using a Subordinate CA Template (as described in https://kb.vmware.com/s/article/2112009 - follow "Creating a new template for vSphere 6.x to use for VMCA as a Subordinate CA")

Export the signed Certificate Chain as a Base64 file (certnew.p7b) and move it to the VCSA /tmp

 

root@mc-vcsa-v-220 [ /tmp ]# openssl pkcs7 -print_certs -in certnew.p7b | sed -n '/---BEGIN CERTIFICATE---/,/---END CERTIFICATE---/p' > vmca_issued_cer.cer
root@mc-vcsa-v-220 [ /tmp ]#

#Check 'modulus' of Public, Private & Cert all match
root@mc-vcsa-v-220 [ /tmp ]# openssl rsa -pubin -in pubkey.pub -modulus -noout | md5sum
9c02b2f12b23f875adc6ed7c2a4ee785  -
root@mc-vcsa-v-220 [ /tmp ]# openssl rsa -in vmca_issued_key.key -modulus -noout | md5sum
9c02b2f12b23f875adc6ed7c2a4ee785  -
root@mc-vcsa-v-220 [ /tmp ]# openssl x509 -in vmca_issued_cer.cer -modulus -noout | md5sum
9c02b2f12b23f875adc6ed7c2a4ee785  -
root@mc-vcsa-v-220 [ /tmp ]#

 

 

Then run up Certificate Manager to Import custom certificate(s) and key(s) to replace existing VMCA Root Signing certificate.

 

root@mc-vcsa-v-220 [ /tmp ]# /usr/lib/vmware-vmca/bin/certificate-manager
 _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
|                                                                     |
|      *** Welcome to the vSphere 8.0 Certificate Manager  ***        |
|                                                                     |
|                   -- Select Operation --                            |
|                                                                     |
|      1. Replace Machine SSL certificate with Custom Certificate     |
|                                                                     |
|      2. Replace VMCA Root certificate with Custom Signing           |
|         Certificate and replace all Certificates                    |
|                                                                     |
|      3. Replace Machine SSL certificate with VMCA Certificate       |
|                                                                     |
|      4. Regenerate a new VMCA Root Certificate and                  |
|         replace all certificates                                    |
|                                                                     |
|      5. Replace Solution user certificates with                     |
|         Custom Certificate                                          |
|         NOTE: Solution user certs will be deprecated in a future    |
|         release of vCenter. Refer to release notes for more details.|
|                                                                     |
|      6. Replace Solution user certificates with VMCA certificates   |
|                                                                     |
|      7. Revert last performed operation by re-publishing old        |
|         certificates                                                |
|                                                                     |
|      8. Reset all Certificates                                      |
|_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _|
Note : Use Ctrl-D to exit.
Option[1 to 8]: 2

Do you wish to generate all certificates using configuration file : Option[Y/N] ? : n

Please provide valid SSO and VC privileged user credential to perform certificate operations.
Enter username [Administrator@vsphere.local]:
Enter password:
certool.cfg file exists, Do you wish to reconfigure : Option[Y/N] ? : n
         1. Generate Certificate Signing Request(s) and Key(s) for VMCA Root Signing certificate

         2. Import custom certificate(s) and key(s) to replace existing VMCA Root Signing certificate

Option [1 or 2]: 2

Please provide valid custom certificate for Root.
File : /tmp/vmca_issued_cer.cer

Please provide valid custom key for Root.
File : /tmp/vmca_issued_key.key

You are going to replace Root Certificate with custom certificate and regenerate all other certificates
Continue operation : Option[Y/N] ? : y
Status : 35% Completed [Replacing Machine SSL Cert...]
Status : 50% Completed [Replace vsphere-webclient Cert...]
Status : 70% Completed [stopping services...]
Status : 85% Completed [starting services...]
Status : 100% Completed [All tasks completed successfully]

root@mc-vcsa-v-220 [ /tmp ]#