This VCSA has been migrated from a Windows vCenter Server 6.5 Update 1 (build-5973321), and then upgraded to vCenter Appliance 6.7 Update 3t (build-22509723), before finally being upgraded to vCenter Appliance 8.0 Update 2b (build-23319993). I also had is PNID changed after it was upgraded to 8.0U2b.
I am trying Certificate Manager v8.0 option #2 (Replace VMCA Root certificate with Custom Signing Certificate and replace all Certificates) to replace the VMCA certificate with a subordinate CA cert from my enterprise signing CA.
Seconds after starting stage #2. I get the error "ERROR certificate-manager ERROR:: INVALID_KEY, the private key doesnot match the certificate. Please provide a valid certificate and Key pair." and it starts a rollback.
I have done this 100's of times with previous versions of vCenter Server over the years and never seen this error.
I notice the default RSA key size has changed to 3072 from 2048 in this version of Certificate Manager, but what else?
Do I need to amend my Windows Subordinate CA template for version 8.0 vCenter Servers? I cannot see an updated KB for this.
Is anyone aware of Certificate Manager 8.0 issues with vCenter 8.0 Update 2b?
I have tried resetting all certs (option #8), which completes OK; but retrying option #2 fails with same message.
I will do some more digging tonight/tomorrow, but would appreciate any feedback if anyone is aware / has also had this issue.
Cheers
M
Something odd is happening with Certificate Manager version 8.0 and/or certool version 2.0.0.
Every CSR generated by them, that is signed by my CA fails to import with the same error.
I tested with a vCenter Appliance 7.0.3 (build-20990077), generated the keys & the CSR using Certificate Manager, signed it with the same CA/Template and it imported & applied without error.
I have worked round my issue on the 8.0 U2b appliance by using OpenSSL to generate the public/private keys, as well as the csr.
*** FOLLOW ON AT YOUR OWN RISK - THIS HAS HAD VERY LIMITED TESTING ***
*** IF YOU HAVE THE SAME/SIMILAR ISSUE AND CAN OPEN AN SR WITH GSS, YOU SHOULD DO SO ***
root@mc-vcsa-v-220 [ /tmp ]# openssl genpkey -algorithm RSA -out /tmp/vmca_issued_key.key -pkeyopt rsa_keygen_bits:4096
...+..+...+.+..+....+......+........+......+.+..+.............+..+.+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++*.......+...+........+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++*.+..+....+.........+...........+.+..+.......+...+..+...+......+............+.........+.+.....+.......+......+......+...+...+...+.....+.......+..+............+.+...+.....+.+..................+.................+...+...+....+........+.+..+..................+.........+.....................+.+...+...+.....+....+.....+.+...+.........+.....+............+...+.+.....+.......+...+...........+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
.....+.+........+....+..+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++*..+...............+...+...+......+....+.........+.....+...+....+...+..+.............+...+.....+.........+..........+......+........+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++*........+..............+....+..+.+........+....+..+................+.....+............+...+.......+...+..+...+.......+.....+....................................+.......+...+.....+.........+............+....+............+..+.+..+.......+.....+.+.....+.+.....+.+........+.+.....................+.....+...+......+......+..................+....+............+...+...+..+.............+..+.+.....+.+...+...+.....+...............+...+.....................+...................+.........+..+.........+.........+...+.........+...+.......+...+........+..........+.....+....+........+.+........................+...+.................+.+............+.........+.........+...........+...+...+....+........+.+..+..................+..............................+....+..+.+..................+..+.......+.....+..........+..................+..+...+...+.+.....................+.....+....+.....+.............+........+.......+...............+........+.......+.....+......+.......+.........+..............+....+.....+......+....+...........+...................+........+...+.+...+.....+......+..........+......+........+...+....+...+........+...............+...+....+...+...+......+..................+...........+.+.....+....+...........+.+..+...+....+...+...........+....+..+...............+...+..........+.......................+.............+......+.....+....+.........+.....+...+...........................................+......+......+...+..+...+.........+.+......+...............+......+.....+...+...................+........+.........+....+..+...+.......+.................+.....................+............+...............+...+....+..+..........+........+...+...+....+............+......+...............+.....+...+............+...+.......+...............+..+...............+.......+..................+.....+.+.....+......+....+..+.......+.................+......+...+....+...............+.....+.......+...+...+............+..+................+.........+......+.....+.........+......+.......+..+...+.+.....................+......+.........+...+..+.........+.+........+......+.......+..+......+.+...+................................+.........+.+...+...+.....+....+......+...+......+..............+.....................+.+...+.........+..+...+..........+.....+..................+.............+......+..................+.....................+..+.....................................+.....+.+.....+....+..................+.....+....+..................+..+.+...+.....+......+..........+.........+..+...+.........+.......+......+..+..............................+......+..........+.....+......+......+....+..............+.........+.+......+..............+..........+............+..+......+......+.+........+......+....+..+....+...........+.......+...........+.+...+..+............................+...+............+........+..........+..+.+.................+...+.......+..+............+.+....................+...+......+...............+.+........................+...+.....+......+....+...........+....+.....+......+...............+.+.....+......+...+.+.........+.........+......+......+.....+.........+.........+..........+..+.+.........+...........+...+.......+.........+...........+...+....+.....+............+......+...+....+...........+......+...+............+.+........+....+.....+..................+..........+.........+...+......+..+............+.......+...+..+..........+......+...+.....+.........+...+....+...+.................+.+..+................+............+.........+........+.....................+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
root@mc-vcsa-v-220 [ /tmp ]# openssl rsa -in /tmp/vmca_issued_key.key -pubout -out /tmp/pubkey.pub
writing RSA key
root@mc-vcsa-v-220 [ /tmp ]# cat << EOF > /tmp/openssl.cnf
[ req ]
default_bits = 4096
encrypt_key = yes
default_md = sha256
utf8 = yes
prompt = no
distinguished_name = req_distinguished_name
req_extensions = v3_intermediate_ca
[ req_distinguished_name ]
countryName = GB
commonName = mc-vcsa-v-220.momusconsulting.com
organizationName = Momus Consulting
organizationalUnitName = Momus Labs
stateOrProvinceName = Hampshire
localityName = Basingstoke
emailAddress = certificate-admin@momusconsulting.com
[ v3_intermediate_ca ]
subjectKeyIdentifier = hash
basicConstraints = critical, CA:true
keyUsage = critical, digitalSignature, cRLSign, keyCertSign
subjectAltName = IP:10.10.1.220, DNS:mc-vcsa-v-220.momusconsulting.com
EOF
root@mc-vcsa-v-220 [ /tmp ]#
root@mc-vcsa-v-220 [ /tmp ]# openssl req -new -key /tmp/vmca_issued_key.key -out /tmp/vmca_issued_csr.csr -config /tmp/openssl.cnf
root@mc-vcsa-v-220 [ /tmp ]#
Sign the CSR on the Enterprise signing CA using a Subordinate CA Template (as described in https://kb.vmware.com/s/article/2112009 - follow "Creating a new template for vSphere 6.x to use for VMCA as a Subordinate CA")
Export the signed Certificate Chain as a Base64 file (certnew.p7b) and move it to the VCSA /tmp
root@mc-vcsa-v-220 [ /tmp ]# openssl pkcs7 -print_certs -in certnew.p7b | sed -n '/---BEGIN CERTIFICATE---/,/---END CERTIFICATE---/p' > vmca_issued_cer.cer
root@mc-vcsa-v-220 [ /tmp ]#
#Check 'modulus' of Public, Private & Cert all match
root@mc-vcsa-v-220 [ /tmp ]# openssl rsa -pubin -in pubkey.pub -modulus -noout | md5sum
9c02b2f12b23f875adc6ed7c2a4ee785 -
root@mc-vcsa-v-220 [ /tmp ]# openssl rsa -in vmca_issued_key.key -modulus -noout | md5sum
9c02b2f12b23f875adc6ed7c2a4ee785 -
root@mc-vcsa-v-220 [ /tmp ]# openssl x509 -in vmca_issued_cer.cer -modulus -noout | md5sum
9c02b2f12b23f875adc6ed7c2a4ee785 -
root@mc-vcsa-v-220 [ /tmp ]#
Then run up Certificate Manager to Import custom certificate(s) and key(s) to replace existing VMCA Root Signing certificate.
root@mc-vcsa-v-220 [ /tmp ]# /usr/lib/vmware-vmca/bin/certificate-manager
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
| |
| *** Welcome to the vSphere 8.0 Certificate Manager *** |
| |
| -- Select Operation -- |
| |
| 1. Replace Machine SSL certificate with Custom Certificate |
| |
| 2. Replace VMCA Root certificate with Custom Signing |
| Certificate and replace all Certificates |
| |
| 3. Replace Machine SSL certificate with VMCA Certificate |
| |
| 4. Regenerate a new VMCA Root Certificate and |
| replace all certificates |
| |
| 5. Replace Solution user certificates with |
| Custom Certificate |
| NOTE: Solution user certs will be deprecated in a future |
| release of vCenter. Refer to release notes for more details.|
| |
| 6. Replace Solution user certificates with VMCA certificates |
| |
| 7. Revert last performed operation by re-publishing old |
| certificates |
| |
| 8. Reset all Certificates |
|_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _|
Note : Use Ctrl-D to exit.
Option[1 to 8]: 2
Do you wish to generate all certificates using configuration file : Option[Y/N] ? : n
Please provide valid SSO and VC privileged user credential to perform certificate operations.
Enter username [Administrator@vsphere.local]:
Enter password:
certool.cfg file exists, Do you wish to reconfigure : Option[Y/N] ? : n
1. Generate Certificate Signing Request(s) and Key(s) for VMCA Root Signing certificate
2. Import custom certificate(s) and key(s) to replace existing VMCA Root Signing certificate
Option [1 or 2]: 2
Please provide valid custom certificate for Root.
File : /tmp/vmca_issued_cer.cer
Please provide valid custom key for Root.
File : /tmp/vmca_issued_key.key
You are going to replace Root Certificate with custom certificate and regenerate all other certificates
Continue operation : Option[Y/N] ? : y
Status : 35% Completed [Replacing Machine SSL Cert...]
Status : 50% Completed [Replace vsphere-webclient Cert...]
Status : 70% Completed [stopping services...]
Status : 85% Completed [starting services...]
Status : 100% Completed [All tasks completed successfully]
root@mc-vcsa-v-220 [ /tmp ]#