Hello,
We have an internal certificate authority (using OpenSSL) and have created a CA intermediate certificate using VCenter's certificate-manager to create the csr.
The certificate is successfully installed using certificate manager (it accepts it without complaining), however, VCenter is not able to start as vpxd is getting a SIGTERM (signal 15) during startup.
I've captured a limited number of vpxd log entries around the signal 15. It appears to be attempting to call a .setCertficate method and this method seems to fail (it is at the end).
I'm working with VCenter 6.7.0.54000
I feel like I may be missing something in the certificate extensions or something to do with certificate creation. Here is the certificate in question.
Certificate:
Data:
Version: 3 (0x2)
Serial Number: 4110 (0x100e)
Signature Algorithm: sha256WithRSAEncryption
Issuer: C = US, ST = Texas, O = "MyCompany, Inc", OU = Development, CN = mydomain.net, emailAddress = frank@mydomain.net
Validity
Not Before: Feb 5 22:45:39 2023 GMT
Not After : Feb 15 22:45:39 2024 GMT
Subject: C = US, ST = Texas, L = Weatherford, O = "MyCompany, Inc", OU = Development, CN = CA
Subject Public Key Info:
Public Key Algorithm: rsaEncryption
RSA Public-Key: (2048 bit)
Modulus:
00:c2:a5:18:82:1f:ba:a9:39:7b:6a:1b:07:90:ff:
bb:ac:a3:75:25:02:23:ed:41:30:01:9d:a2:12:94:
c7:b3:83:d1:be:1d:9d:d1:e5:87:4e:7a:61:70:16:
c3:3f:d1:d1:1f:8f:93:59:a0:01:1f:e1:56:68:ab:
78:42:40:03:fe:cb:4f:d5:fb:73:a8:42:9f:fb:74:
82:0f:2e:9b:be:83:67:2c:9e:0b:55:31:ee:32:0c:
19:ff:16:c4:3b:7a:d0:c3:94:66:a1:29:02:bb:13:
58:29:04:27:a9:72:50:7b:a0:a0:6c:8c:a6:79:42:
62:ca:db:be:4e:d4:a0:9c:be:89:68:29:bd:87:0e:
04:65:7a:1b:36:ce:d4:17:bc:97:c2:1b:ce:d2:18:
b2:b3:b2:9a:7a:f1:dd:90:fc:82:4b:ba:30:be:69:
4c:16:90:85:86:1b:b7:a6:ba:92:4b:88:af:ec:f2:
76:0f:6d:d3:0e:8f:93:83:1e:03:52:03:33:94:17:
03:7b:88:b0:9d:ae:5a:5a:c5:d8:ea:b7:72:86:4b:
14:f2:8b:3b:4f:8a:59:d1:8f:82:ab:8b:8a:40:28:
11:ea:34:90:2b:c7:c8:f7:d1:61:d4:a8:ae:6d:a7:
e8:ed:58:a3:d1:52:d4:8b:22:1b:51:ce:05:95:92:
1c:e9
Exponent: 65537 (0x10001)
X509v3 extensions:
X509v3 Subject Key Identifier:
6E:81:0D:26:2C:A6:D9:A5:11:B8:01:7C:EE:D3:5A:AB:85:C4:2D:C1
X509v3 Authority Key Identifier:
keyid:F3:9C:8D:DB:28:A1:8E:CB:2D:30:58:7F:DF:9F:FB:98:64:5A:1B:A6
X509v3 Basic Constraints: critical
CA:TRUE
X509v3 Key Usage: critical
Digital Signature, Non Repudiation, Certificate Sign, CRL Sign
Signature Algorithm: sha256WithRSAEncryption
1a:d7:4d:ca:a6:46:cc:91:11:da:10:6c:93:c3:db:0c:11:7d:
6c:6e:94:d6:56:87:68:b8:4b:6b:4d:01:e0:a7:49:1c:e7:64:
49:d1:4e:65:6d:13:1e:74:72:c1:c4:6b:59:5d:8f:dc:35:33:
1b:bb:93:5f:6b:a4:ea:9a:05:9b:95:49:cf:39:e4:f4:c1:33:
d1:6e:13:a6:6f:7e:c9:d6:cb:db:1f:48:5a:05:a7:e0:4e:85:
87:7e:05:29:fe:49:58:58:e7:f5:a1:14:35:9c:88:6c:06:00:
ae:64:ae:24:75:95:17:9f:0c:77:bd:0d:a7:0a:63:e0:4d:13:
91:47:9a:2c:e7:b8:54:5c:91:72:b2:a4:95:c6:e9:48:4f:db:
02:f2:c5:a7:2d:68:f2:dc:88:52:4e:f4:71:9d:5c:06:10:50:
a3:ed:a5:9f:06:07:89:f0:bb:69:c4:e2:2d:23:d2:9f:34:bc:
af:36:b9:28:62:1c:2a:a3:f7:ad:cd:36:c2:15:54:a7:87:d6:
58:6b:d2:93:67:20:f5:d5:35:06:bf:c3:89:e8:1b:06:4e:d2:
1e:99:ce:5f:8e:b6:fa:54:6d:bd:f6:de:01:cc:2e:81:82:da:
6d:d7:5b:fd:03:92:c6:b1:60:aa:32:3c:c3:c8:43:c0:6c:86:
7b:03:b7:fe:99:91:b7:fb:25:2a:a3:54:f1:51:dd:46:cf:57:
3a:c9:46:64:0f:ad:83:08:be:e7:66:51:63:f4:90:f2:ac:65:
05:c7:d6:72:87:fc:3f:f4:1a:86:5a:68:e9:9c:68:dc:0d:4e:
e1:57:df:6c:00:41:0b:68:62:95:85:c0:ff:e9:05:81:67:2c:
8e:a1:88:7b:3b:88:ca:25:bc:2e:b6:8f:49:0c:fa:d9:e0:47:
d6:8a:e8:8f:85:ed:bb:e6:df:43:15:37:a8:60:6d:dc:43:48:
ee:42:b4:9a:56:cb:35:98:9c:70:99:24:49:dd:dc:1b:41:70:
f9:aa:27:bc:6d:fe:9b:2b:08:e2:f7:e2:ac:d3:df:aa:43:8c:
00:de:a9:32:c4:02:bf:0d:f9:0e:c4:69:5b:0a:a3:38:1e:1a:
14:ba:8e:6f:cc:37:e9:ac:5b:9e:54:6f:9b:64:1e:17:fb:ed:
28:d6:60:76:f5:f3:c5:11:f6:2b:11:72:1d:af:36:4c:aa:02:
e8:31:4f:50:21:ff:86:f1:a4:6f:16:80:ae:1f:3e:11:ec:80:
95:61:f2:96:3c:b9:e2:21:a2:d7:53:57:0e:8c:f2:d5:56:fa:
74:23:3c:a9:52:f8:d0:d1:9a:db:d3:99:95:11:02:f1:77:97:
03:82:6e:54:46:da:f5:48
2023-02-05T23:26:41.517Z info vpxd[31590] [Originator@6876 sub=AuthorizeManager opID=31155621] [Auth]: User VSPHERE.LOCAL\Administrator
2023-02-05T23:26:41.518Z info vpxd[31590] [Originator@6876 sub=vpxLro opID=31155621] [VpxLRO] -- FINISH lro-8488
2023-02-05T23:26:41.533Z info vpxd[31564] [Originator@6876 sub=vpxLro opID=361c547] [VpxLRO] -- BEGIN lro-8490 -- ExtensionManager -- vim.ExtensionManager.setCertificate -- 52d54268-3287-0396-bf36-4f316291e435(52fbdd5e-5e93-4043-9dd0-ba942a6f5623)
2023-02-05T23:26:41.534Z info vpxd[31564] [Originator@6876 sub=vpxLro opID=361c547] [VpxLRO] -- FINISH lro-8490
2023-02-05T23:26:41.534Z info vpxd[31564] [Originator@6876 sub=Default opID=361c547] [VpxLRO] -- ERROR lro-8490 -- ExtensionManager -- vim.ExtensionManager.setCertificate: vim.fault.NotFound:
--> Result:
--> (vim.fault.NotFound) {
--> faultCause = (vmodl.MethodFault) null,
--> faultMessage = <unset>
--> msg = ""
--> }
--> Args:
-->
--> Arg extensionKey:
--> "com.vmware.imagebuilder"
--> Arg certificatePem:
--> "-----BEGIN CERTIFICATE-----
--> MIIEGDCCAwCgAwIBAgIJAOFnQF8bcuFzMA0GCSqGSIb3DQEBCwUAMHUxCzAJBgNV
--> BAYTAlVTMQ4wDAYDVQQIDAVUZXhhczEUMBIGA1UEBwwLV2VhdGhlcmZvcmQxHTAb
--> BgNVBAoMFEFlcmlhbCBSb2JvdGljcywgSW5jMRQwEgYDVQQLDAtEZXZlbG9wbWVu
--> dDELMAkGA1UEAwwCQ0EwHhcNMjMwMjA1MjMxNjM4WhcNMjQwMjE1MjI0NTM5WjCB
--> iTEXMBUGA1UEAwwOdnB4ZC1leHRlbnNpb24xFzAVBgoJkiaJk/IsZAEZFgd2c3Bo
--> ZXJlMRUwEwYKCZImiZPyLGQBGRYFbG9jYWwxCzAJBgNVBAYTAlVTMTEwLwYDVQQL
--> DChtSUQtMTU1ZTg4ZWMtNjcxOC00ZjYyLWE4NzEtMTM4MTI4OTU0Njc2MIIBIjAN
--> BgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEAtt8IQhXhfuVlb8g8xu8yRvsYkrwn
--> AhJMCyViM74QeQ47K0TioguDV8wm/zDN5kma97AQFKZ/bGNcisHUV14qoX2MUmnr
--> 5ntv9BGztV9te7NacW0GqcnDxEDnS3+Lobetl9eQnSXMeiz+mvZYSJ/opHlVL/q1
--> BKy5a9By4Q9tdPS7pOEvr+K6W97UX1Xje1G7UK1mfhl9EGKcj3o/GJvXwFPEBdtx
--> DVnJvgc+ldsEclpdkT4xiTIiBBuVJu3g4Sx7eHpssu6fZSlvWS9tZIF6n5je/Mng
--> L6DHLc7RWlg4kmBN9btxJ1FNd34lnXAzjbDjLPWRGIioDyS4zFhXwNBjmwIDAQAB
--> o4GVMIGSMAsGA1UdDwQEAwIF4DAdBgNVHQ4EFgQUh4jODNakYb6VkfnDFZxwH6+K
--> xYgwHwYDVR0jBBgwFoAUboENJiym2aURuAF87tNaq4XELcEwQwYIKwYBBQUHAQEE
--> NzA1MDMGCCsGAQUFBzAChidodHRwczovL3ZjZW50ZXIuYXJpbGFicy5uZXQvYWZk
--> L3ZlY3MvY2EwDQYJKoZIhvcNAQELBQADggEBAJ7DQbWj7aE4uTqmB8UbG4rAQLQZ
--> 4JUWdq3F4ztloWEwOaZygcXzVdCg7+G1CeCl+bX/oewqW8h5HB9npBeKjBiVTi+K
--> 1fcCDfuABna8UaQOCNrKWxO3dpk8+jRtVQ8ykuwPVytr7vryww9G45Aa57q5ee0B
--> w7pKvLT7Plxax1EKEF2pdgTWc7MgX4xdIa5+5p91vMD3JDXiwGa2XTnV7gU0wg7p
--> 3S4Ph7kU3BffxrvymTTi4OPKNBobdKlCYZd31Ap0P1ql7mV4d7nWi4nwYOcn9rvb
--> STbDNf8BsSdq2+FAw5/jAnHqG9QrOUkH3jIcjo9/NBTnKKEy3wR3w+HOsDc=
--> -----END CERTIFICATE-----
--> -----BEGIN CERTIFICATE-----
--> MIIE3zCCAsegAwIBAgICEA4wDQYJKoZIhvcNAQELBQAwgYoxCzAJBgNVBAYTAlVT
--> MQ4wDAYDVQQIDAVUZXhhczEdMBsGA1UECgwUQWVyaWFsIFJvYm90aWNzLCBJbmMx
--> FDASBgNVBAsMC0RldmVsb3BtZW50MRQwEgYDVQQDDAthcmlsYWJzLm5ldDEgMB4G
--> CSqGSIb3DQEJARYRa2V2aW5AYXJpbGFicy5uZXQwHhcNMjMwMjA1MjI0NTM5WhcN
--> MjQwMjE1MjI0NTM5WjB1MQswCQYDVQQGEwJVUzEOMAwGA1UECAwFVGV4YXMxFDAS
--> BgNVBAcMC1dlYXRoZXJmb3JkMR0wGwYDVQQKDBRBZXJpYWwgUm9ib3RpY3MsIElu
--> YzEUMBIGA1UECwwLRGV2ZWxvcG1lbnQxCzAJBgNVBAMMAkNBMIIBIjANBgkqhkiG
--> 9w0BAQEFAAOCAQ8AMIIBCgKCAQEAwqUYgh+6qTl7ahsHkP+7rKN1JQIj7UEwAZ2i
--> EpTHs4PRvh2d0eWHTnphcBbDP9HRH4+TWaABH+FWaKt4QkAD/stP1ftzqEKf+3SC
--> Dy6bvoNnLJ4LVTHuMgwZ/xbEO3rQw5RmoSkCuxNYKQQnqXJQe6CgbIymeUJiytu+
--> TtSgnL6JaCm9hw4EZXobNs7UF7yXwhvO0hiys7KaevHdkPyCS7owvmlMFpCFhhu3
--> prqSS4iv7PJ2D23TDo+Tgx4DUgMzlBcDe4iwna5aWsXY6rdyhksU8os7T4pZ0Y+C
--> q4uKQCgR6jSQK8fI99Fh1Kiubafo7Vij0VLUiyIbUc4FlZIc6QIDAQABo2MwYTAd
--> BgNVHQ4EFgQUboENJiym2aURuAF87tNaq4XELcEwHwYDVR0jBBgwFoAU85yN2yih
--> jsstMFh/35/7mGRaG6YwDwYDVR0TAQH/BAUwAwEB/zAOBgNVHQ8BAf8EBAMCAcYw
--> DQYJKoZIhvcNAQELBQADggIBABrXTcqmRsyREdoQbJPD2wwRfWxulNZWh2i4S2tN
--> AeCnSRznZEnRTmVtEx50csHEa1ldj9w1Mxu7k19rpOqaBZuVSc855PTBM9FuE6Zv
--> fsnWy9sfSFoFp+BOhYd+BSn+SVhY5/WhFDWciGwGAK5kriR1lRefDHe9DacKY+BN
--> E5FHmiznuFRckXKypJXG6UhP2wLyxactaPLciFJO9HGdXAYQUKPtpZ8GB4nwu2nE
--> 4i0j0p80vK82uShiHCqj963NNsIVVKeH1lhr0pNnIPXVNQa/w4noGwZO0h6Zzl+O
--> tvpUbb323gHMLoGC2m3XW/0DksaxYKoyPMPIQ8BshnsDt/6Zkbf7JSqjVPFR3UbP
--> VzrJRmQPrYMIvudmUWP0kPKsZQXH1nKH/D/0GoZaaOmcaNwNTuFX32wAQQtoYpWF
--> wP/pBYFnLI6hiHs7iMolvC62j0kM+tngR9aK6I+F7bvm30MVN6hgbdxDSO5CtJpW
--> yzWYnHCZJEnd3BtBcPmqJ7xt/psrCOL34qzT36pDjADeqTLEAr8N+Q7EaVsKozge
--> GhS6jm/MN+msW55Ub5tkHhf77SjWYHb188UR9isRch2vNkyqAugxT1Ah/4bxpG8W
--> gK4fPhHsgJVh8pY8ueIhotdTVw6M8tVW+nQjPKlS+NDRmtvTmZURAvF3lwOCblRG
--> 2vVI
--> -----END CERTIFICATE-----
--> -----BEGIN CERTIFICATE-----
--> MIIGDDCCA/SgAwIBAgICEAAwDQYJKoZIhvcNAQELBQAwgZ4xCzAJBgNVBAYTAlVT
--> MQ4wDAYDVQQIDAVUZXhhczEUMBIGA1UEBwwLV2VhdGhlcmZvcmQxHTAbBgNVBAoM
--> FEFlcmlhbCBSb2JvdGljcywgSW5jMRIwEAYDVQQLDAlEZXZlbG9wZXIxFDASBgNV
--> BAMMC2FyaWxhYnMubmV0MSAwHgYJKoZIhvcNAQkBFhFrZXZpbkBhcmlsYWJzLm5l
--> dDAeFw0yMzAyMDUwNDQ5MzRaFw0zMzAyMDIwNDQ5MzRaMIGKMQswCQYDVQQGEwJV
--> UzEOMAwGA1UECAwFVGV4YXMxHTAbBgNVBAoMFEFlcmlhbCBSb2JvdGljcywgSW5j
--> MRQwEgYDVQQLDAtEZXZlbG9wbWVudDEUMBIGA1UEAwwLYXJpbGFicy5uZXQxIDAe
--> BgkqhkiG9w0BCQEWEWtldmluQGFyaWxhYnMubmV0MIICIjANBgkqhkiG9w0BAQEF
--> AAOCAg8AMIICCgKCAgEAxfbhYycv83WbfkaQOwS3kVyTzanX3bbQvqPgydgi2zag
--> 8Af3IxQsO3raSu+9hQL5IluKE2/8jL7ZDiv+wSe7H+9PvckZFCufkaN+Zp6lP703
--> nX7e6hPnUnwXLbqMPtfaLDWQ9UF1uKkrDvfXWwtDSQytTYR1w2Cmb5IFDnPhx27/
--> 82WTAfwIZZI4tNL9JDJZ3RZUsL/SLwH20mjkBkyKELv6ScQb58FquxONabAOOQWs
--> 2DcMUi6zuiR7ruoWUpi3Xga+SMmPwbP847e6PvX+r43koIWELZfyeMf+MkmpMNOV
--> P5sS09xa9zWr3L6e9KCGZkOXrhw/ktIIhR1FpYv4Dmj3gWFTCPHLVCfJ4AaQHbt7
--> XKkJVIP6o9b0RnpSsxubPWNlXUrIkd+RRgkZCWRcxrJBA+SBBCXwTWDZLa2qGHUp
--> 0iTF8pWv1z19pBq6NTqnVXzn+WOSRoplzzu0a+I01ShPzDjUwW2ERX6WmYv0cpI2
--> uMsWCcyY/hOuGYsUtBP2RTPzomNkdzjcU9KwQwPwwaQEjpvQ2n+f3R71xSmTbW6F
--> /YmIyClazBMz4gyYS9f8Gl1wf9xrWV7wZmbuJro2BnEVr9QbCzq1QyY0mAMICcGo
--> 7Y2uaZHWCpPDsoTrprKy3WfBjKS9ecoTjPg9MuKLcmGS6RgmVJ9IbQzMCowd+u0C
--> AwEAAaNmMGQwHQYDVR0OBBYEFPOcjdsooY7LLTBYf9+f+5hkWhumMB8GA1UdIwQY
--> MBaAFD0BZs0nTOmKikVxfHYYzOo4NcUCMBIGA1UdEwEB/wQIMAYBAf8CAQAwDgYD
--> VR0PAQH/BAQDAgGGMA0GCSqGSIb3DQEBCwUAA4ICAQBU8QJfWOMKrsGk71RMOXU2
--> LjsBIPpybLO4KaxIdl0wiDgJ/T2oCEUPAsejyUJKPOrr/bt84cqfvGx/ou0yhBq8
--> u0WsnjQI1zwG6Mcfcrmz1AFO8ewQBZWLoavkthMwEhrZ2T7s4+hFXf6iIyhlLdqY
--> zjC/3Jk+ivm33CfPC5WOsgy37iEjonXXCvZVPBHqHb3gX3DPW4YENiielBdjwEaG
--> Yyasbue64Z4ZsOxmkixhNSDsWCHoG4wGEglbYsyxhDWZ3kMttxGT4F1yjfxi+Fiq
--> B1puP5K56A3Iq6axhC8VsltdyBMSVpr3kvEKKKsMNvpbzb0INLcQ7zxs7AZd/kl6
--> 2tU9sx0+f3wqfijmc3FFH8wCKoZMRfBZuLAtdA8aIjh7vZN3p76xJ+7ea1wrRmdQ
--> oRcPSnFCWK8xB4FyZF9WBG57ct0kwDzqLwuBCEUXLp9RRbC1cz8EdvOmXdr1FeUB
--> wNDgJ6zxKQLalinnuqJunjSYMBP/7AX/rH/XAOODoQnGtiQzJEwLB95HZoUxJYey
--> XhETO84H9DZVy6rsYPPV14NvHDyTlQ9ejlZ8yxyRt1cQQNgBvH3dftBxem1VM6bf
--> zv4PzbFS2jdS+GDhYTtZNoppAxZr+isukehRBW102j/Ij7M8vZb281ZKtM5MIIag
--> HJXeakWpFb+2AATP2TkKrw==
--> -----END CERTIFICATE-----
-->
--> "
2023-02-05T23:26:41.537Z info vpxd[31605] [Originator@6876 sub=vpxLro opID=7deba0d] [VpxLRO] -- BEGIN lro-8491 -- SessionManager -- vim.SessionManager.logout -- 52d54268-3287-0396-bf36-4f316291e435(52fbdd5e-5e93-4043-9dd0-ba942a6f5623)
2023-02-05T23:26:41.537Z info vpxd[31605] [Originator@6876 sub=vpxLro opID=7deba0d] [VpxLRO] -- FINISH lro-8491
2023-02-05T23:26:43.006Z info vpxd[31565] [Originator@6876 sub=vpxdvpxdSignal] Signal 15 received, exiting
2023-02-05T23:26:43.006Z info vpxd[31565] [Originator@6876 sub=Default] Initiating VMware VirtualCenter shutdown
2023-02-05T23:26:43.006Z info vpxd[31487] [Originator@6876 sub=Default] Shutting down VMware VirtualCenter
Any help is appreciated.
Thank you,
Kevin
It means that you have uploaded the wrong certificate.
Are you using 3rd party Certificate provider or Microsoft Certificate Authority?
From where you have requested CSR? via GUI or via SSH?
I used Certificate Manager (via ssh) to get the CSR. It creates a CSR and a private key.
I copied the CSR to our internal certificate authority (it uses openssl) and create the signed CA certificate (CA:TRUE, signed by an intermediate key).
I concatenate the new certificate, intermediate cert and the root cert into a single file (in that order) and copy it back to the vcenter server.
I then use certifiate-manager (option 2) to import the certificate, combining it with the private key.
It accepts it (as long as I wait long enough, there is that Start Time error if I do it too soon) and gets to about 85% (that is when vpxd decides to give up).
This is the command used to create the certificate:
openssl ca -config intermediate/openssl.cnf -extensions v3_ca -days 375 -notext -md sha256 -in intermediate/csr/vmca_issued_csr.csr -out intermediate/certs/vmca.certs.pem
v3_ca extensions contain the following:
[ v3_ca ]
# Extensions for a CA
subjectKeyIdentifier = hash
authorityKeyIdentifier = keyid:always,issuer
basicConstraints = critical, CA:true
keyUsage = critical, digitalSignature, cRLSign, keyCertSign, nonRepudiation
I'm pretty sure I'm importing the correct certificate (well, at least the certificate I intend to import, whether it is correct or not is another matter...)
Thanks for taking a look at this.
Kevin
Check that article
https://virtualblog.nl/2020/10/26/vmware-vcenter-replace-machine-certificate-with-custom-ca/
here is step-by-step guide. Maybe you missed smth.
Fingers crossed
Great article! Thank you. I've scoured the internet looking for other perspectives. Found some things that did help, but this by far the most complete I've seen. Thanks again
The only thing is that I really need to figure out how to replace the CA certificate (the Certificate Authority part of VCenter , Option 2 of Configuration Manager). This is what is failing. I've been able to replace the MachineSSL certificate, but not the built in CA Certificate.
Thanks,
Kevin