I'm trying to replace all certs used by vCenter 5.1 and have so far succeeded using the predeployment method stated in Derek's blog here. However, this link, and the corresponding VMware (threadbare) documentation for SSL certificate replacement do not address ESXi Dump Collector, Syslog Collector, Auto Deploy, and Authentication Proxy.
The only documentation I can find is in vCenter Setup guide page 76:
---
You can use your own Certificate Authority (CA) by replacing the Open SSL certificate (rbdca.crt) and the OpenSSL private key (rbd-ca.key) with your own certificate and key file.
On Windows, the files are in the SSL subfolder of the Auto Deploy installation directory. For example, on Windows 7 the default is c:\ProgramData\VMware\VMware Auto Deploy\ssl.
---
My question is why does Auto Deploy need the CA certificate and private key in addition to the standard rui.crt and rui.key for SSL usage?
Also, what are the requirements for these certificates and for the other vCenter addons (Syslog Collector, etc...)? Can the same certificate/key used by vCenter be used here for these applications since they reside on the same server as vCenter?
As noted in Derek's blog, the certificates used by Inventory Service, Update Manager, vCenter Server, SSO, and the Web client must all contain unique OU= fields, but do those restrictions apply to these modules as well?
-Ed
Not sure whether you’ve made progress since your post, but for what it’s worth here’s what I’ve been able to figure out.
I’ve replaced the “rui” certificates for Dump Collector, Syslog Collector and Auto Deploy using Derek Seaman and others’ techniques. I used unique OU names, but suspect most of them would work with generic names, as they don’t integrate with SSO (yet.) Autodeploy and Authentication Proxy are special cases. I'm still working on Authentication Proxy.
As for Autodeploy: Autodeploy comes with two SSL certificates, rbd-ca and rui.
From my experience, rui can be replaced in the same way other vCenter components are updated. It's used to render the (minimal) GUI with its Boot and Log download links, and to serve content when hosts boot. (Edit: you'll need to re-register Autodeploy with vCenter, see response later in this thread.)
rbd-ca is the CA certificate used to issue host certificates for hosts provisioned with autodeploy and thus has different properties for the “normal” rui-type certificates. Implementing a CA in this way allows vCenter to allocate a persistent certificate to stateless hosts on first boot; if each host auto-generated its own certificate on every boot, certificate validation would fail when vCenter attempts to reconnect to the host.
Host certificates issued by rbd-ca are stored in the Autodeploy SSL folder, with each host having a subfolder (e.g. host-28, host-143, etc.) under the SSL folder. On Windows, this is in C:\ProgramData\VMware\VMware vSphere Auto Deploy\ssl. This certificate is passed to each host every time it boots up.
The .srl file associated with the rbd-ca certificate in the SSL folder is just a serial number file. It can be left intact when replacing the CA certificate. Autodeploy automatically updates this file each time it issues a certificate, using it to ensure each certificate has a unique identifier.
To generate a new rbd-ca certificate, you will need to either create a new CA, or get your CA to issue a new subordinate CA certificate. I did it via an existing OpenSSL CA I use purely for my lab environment (“here’s one I prepared earlier”) so it was relatively straight-forward for me, but you will have difficulty doing this if you do not have direct control of the issuing CA. A subordinate CA that can issue its own certificates has significant security implications - particularly since the rbd-ca key is stored unencrypted in this case - so should not be done without proper consideration.
By no means a guide, but to point anyone doing this via OpenSSL in the general direction I followed to issue a suitable subordinate CA:
[ v3_ca ]
…
basicConstraints = CA:true,pathlen:0
…
keyUsage = keyCertSign
…
OpenSSL genrsa 2048 -nodes -config %OpenSSL_Conf% > rbd-ca.key
OpenSSL req -out rbd-ca.csr -key rbd-ca.key -new -config %OpenSSL_Conf%
OpenSSL ca -config %OpenSSL_Conf% -days 3652 -extensions v3_ca -out rbd-ca.crt
My Autodeploy hosts in the lab are now issued automatically with certificates I trust No need for complex replacement procedures after-the-fact.
For the rui certificate, you would have used the same procedure but would have used usr_cert in stead of v3_ca in the command above. Also note that I already set the location for my CA inside the OpenSSL config, so none of that is passed on the command line in the examples above.
Ideally, after you’ve replaced the rbd-ca cert, you would trust the root CA (Windows: Trusted Root Certification Authorities) as well as the rbd-ca (Windows: Intermediate Certificate Authorities) on the clients where you’re accessing your vCenter and hosts from. This prevents the nasty certificate warning the GUI/browser presents.
Message was edited by: jhanekom (added note about Autodeploy re-registration when replacing rui)
PS - just remembered that after replacing the Autodeploy rui certificate, you will need to run the following command:
"C:\Program Files (x86)\VMware\VMware vSphere Auto Deploy\autodeploy-register.exe" -R -a <vcenter server IP> -u Username -w Password -s "C:\ProgramData\VMware\VMware vSphere Auto Deploy\vmconfig-autodeploy.xml"
Note: this work was done around vSphere 5.0. Will double-check results with vSphere 5.1 but I suspect it will be similar.
I've done some work around the vSphere Authentication Proxy now; still not confident I've got it down pat, but I have some interesting observations that might be useful to someone out there.
The first step with Authentication Proxy is to replace the rui certificate, as with the other services. The cert resides in C:\ProgramData\VMware\vSphere Authentication Proxy\ssl. As with Autodeploy, I had to run the following command after replacing the certificate:
"C:\Program Files (x86)\VMware\vSphere Authentication Proxy\cam-register.exe" -r -a <vcenter server IP> -u Username -p Password -s "C:\ProgramData\VMware\vSphere Authentication Proxy\vmconfig-cam.xml"
When you restart the Proxy Adapter service, you should see some content appearing in C:\ProgramData\VMware\vSphere Authentication Proxy\logs\camadapter.log, with no error messages. Amongst the entries, you should see it deleting the certificate in IIS, and re-adding it. It does this on every startup. Go ahead and verify that you can actually see the new certificate in IIS (edit the SSL binding of the site and view the cert - or just browse to https://<vcenter>:51915/ssl (should get a successful SSL connection, with a "You do not have permission..." message displayed on screen.)
Unlike the other services, Authentication Proxy has an affinity for a combination of DER-encoded certificates. The VMware documentation seems to call this "Windows format", though I'm not sure where the authors got that idea from. To cover the bases, the other certificates in use by Authentication Proxy should all be in both "CRT" (base64-encoded) and DER (non-base64-encoded) format. The service seems to automatically create DER-encoded copies of the ca.crt certificate, but not the certificates in the "trust" folder.
As expected, there appears to be some interaction between Authentication Proxy and Autodeploy. The default Authentication Proxy "ca.der" certificate in the Authentication Proxy SSL folder is the same one used by Autodeploy. I'm not quite sure yet how these two services deal with different installation orders etc. I'm also not entirely sure how this ca.der certificate is used, but I suspect it's to allow individual hosts to validate the certificate chain that signed the Authentication Proxy cert. In my case, I replaced it with the custom CA certificate I created for my Autodeploy instance above; this worked. After some testing, it seems that using the root CA cert here worked as well.
A third building block is the C:\ProgramData\VMware\vSphere Authentication Proxy\trust folder. According to the documentation, this folder should contain the Authentication Proxy certificate if you replace it with a custom-signed cert. This didn't make much sense to me, so I placed my CA certificate in there. This worked; either my Root CA or the Autodeploy CA worked, presumably because this allows the Authentication proxy to trust the host certs issued by Autodeploy. Leaving the trust folder empty caused Authentication Proxy to "break" - the service started, but hosts would throw an "access denied" message if you attempted to join them to the domain. Note that after you restart the Proxy Adapter service, you should now see a message in camadapter.log along the lines of "Add trust certificate <folderpath>"
Finally, it seems that the Authentication Proxy automatically retrieves the certificates from each host connected to vCenter and places them in the ssl folder. It does this the moment a new host is connected. For example, if you join a new host, you might see a new file called "host-28.der/host-28.crt" appear in the folder. I suspect it uses this certificate to determine if it's a legitimate host, and that this somehow ties up to one of the two CA certificates in the ssl and trust folders.
So, in summary, I did the following, which seems to work:
As I said, I'm not 100% confident I've got it fully figured out, but the above seems to work for me so far - my hosts are issued with trusted certificates automatically by AutoDeploy, and I can join them to the domain using the Authentication Proxy service (which now uses a trusted cert itself) without importing any certificates.
Thanks to information provided by Derek’s blog and jhanekom’s post I’ve put together the steps to use a Microsoft Enterprise Subordinate CA to issue new certificate to AutoDeployed hosts and steps to replace the AutoDeploy vCenter plug-in certificate.
Test lab environment:
VM1 – Windows 2008R2 SP1, Domain Controller, Microsoft Root CA
VM2 - Windows 2008R2 SP1, vCenter 5.1b, Microsoft Subordinate CA
C:\ProgramData\VMware\VMware vSphere Auto Deploy\ssl\<ESXi_host>\rui.crt
Thanks Todd - excellent bit of work putting together something that is practically usable!
Perhaps a bit unrelated to this thread, but I think I've found a bug in 5.1 where the CAM Adapter service will truncate the host "identity" name to 16 characters when it attempts to set up the SSL Certificates in IIS using netsh.
In vmconfig-cam.xml, if serviceAddress in the Config section is longer than 16 characters, you will get an error message in camadapter.log. For example, for a host call vcenter01.lab.local, you'll see this:
2013-03-13 18:13:12: Execute cmd /c netsh http delete sslcert ipport=vcenter01.lab.lo:51915 in path C:\Program Files (x86)\VMware\vSphere Authentication Proxy\
2013-03-13 18:13:12: Failed to bind CAM website with CTL
2013-03-13 18:13:12: Failed to initialize CAMAdapter.
This appears to happen regardless of whether you replace SSL Certificates. I'm not sure whether this happened in 5.0 as well.
If I set it to use the IP address, things work properly and it looks like this:
2013-03-13 18:26:50: Execute cmd /c netsh http delete sslcert ipport=172.16.1.141:51915 in path C:\Program Files (x86)\VMware\vSphere Authentication Proxy\
2013-03-13 18:26:51: Execute cmd /c netsh http add sslcert ipport=172.16.1.141:51915 certhash=f1251b7bd947a2c19bc9547151320da78608343e appid={687bf1d4-d3f8-443e-931a-7c0f7ffb658e} sslctlidentifier={B0A90AB6-8F8F-4692-9652-23F6404E143B} sslctlstorename=CA clientcertnegotiation=enable in path C:\Program Files (x86)\VMware\vSphere Authentication Proxy\
2013-03-13 18:26:51: Initialize Monitor
2013-03-13 18:26:51: Start Monitor
2013-03-13 18:26:51: Add certificate: host-45
2013-03-13 18:26:51: Certificate is signed by some CA.
Workarounds:
I'll try to get around to file a bug report sometime.
Bit of Googling indicates Kenneth van Ditmarsch found this 4 months ago already 🙂 http://virtualkenneth.com/2012/11/19/vsphere-authentication-proxy-failed-to-bind-cam-website-with-ct...
I tried your procedure to replace the Authentication Proxy certificate with no luck. I did successfully replace the AutoDeploy certificates and my subordinate CA is issuing certificates to my ESXi hosts. When I restart the Authentication Proxy Adapter the camadapter log shows Failed to bind CAM website with CTL then Failed to initialize CAMAdapter. If you have more details that would be greatly appreciated.
Thanks.
This procedure work perfectly for me.
I use this method,Why Windows Event Log show:
The description for Event ID 1 from source AutoDeploy cannot be found. Either the component that raises this event is not installed on your local computer or the installation is corrupted. You can install or repair the component on the local computer.
If the event originated on another computer, the display information had to be saved with the event.
The following information was included with the event:
[9168]vc_monitor:WARNING:Current VC(id: 1, attr: vc01.corp.demo.com, instance: 17233E11-13DC-4C08-AF6D-F159CA523962) currentTime : 2015-09-16 04:20:57.113775+00:00
the message resource is present but the message is not found in the string/message table
The description for Event ID 1 from source AutoDeploy cannot be found. Either the component that raises this event is not installed on your local computer or the installation is corrupted. You can install or repair the component on the local computer.
If the event originated on another computer, the display information had to be saved with the event.
The following information was included with the event:
[9168]vc_extension:ERROR:Using https://vc01.corp.demo.com:6502 as the managementUrl
the message resource is present but the message is not found in the string/message table
Thank You
Have you checked whether you perhaps have this issue: http://virtualkenneth.com/2012/11/19/vsphere-authentication-proxy-failed-to-bind-cam-website-with-ct...
Regardless, you should investigate C:\ProgramData\VMware\vSphere Authentication Proxy\logs\camadapter.log. The little bit that gets pushed by vCenter to the Windows Event log tends to be extremely high-level.
I see the log is show: but I don't know what it means.:smileyconfused:
2015-09-16 18:57:11,279 [6996]INFO:dbsetup:dbsetup: dbapi_version=2.4.1; sqlite_version=3.7.10
2015-09-16 18:57:11,279 [6996]DEBUG:dbsetup:installed DB connection <sqlite3.Connection object at 0x041DCDA0> for thread <_DummyThread(Dummy-1, started daemon)>
2015-09-16 18:57:11,293 [6996]INFO:dbsetup:Finished VACUUMING the database.
2015-09-16 18:57:11,319 [6996]INFO:dbsetup:setting database defaults.
2015-09-16 18:57:11,328 [6996]INFO:dbsetup:Performing the integrity check on DB
2015-09-16 18:57:11,329 [6996]INFO:dbsetup:Integrity check results --
ok
2015-09-16 18:57:11,329 [6904]INFO:rbd_watchdog_windows:starting Maintenance
2015-09-16 18:57:11,331 [6948]INFO:rbd_watchdog_windows:starting Proxy
2015-09-16 18:57:11,332 [6948]INFO:log:Twisted logging enabled
2015-09-16 18:57:11,332 [6996]INFO:cacher:BEGIN validating cache contents
2015-09-16 18:57:11,332 [6948]INFO:rproxy:static image -- table-bg.gif
2015-09-16 18:57:11,334 [6948]INFO:rproxy:static image -- deploy16x.png
2015-09-16 18:57:11,334 [6948]INFO:rproxy:static image -- deploy32x.png
2015-09-16 18:57:11,335 [6948]INFO:rproxy:static image -- favicon.ico
2015-09-16 18:57:11,336 [6948]INFO:rproxy:listening on IPv6
2015-09-16 18:57:11,338 [6948]INFO:netcat:added netcat acceptor -- (23, 1, 0, '', ('::', 6501, 0, 0))
2015-09-16 18:57:11,338 [6948]INFO:netcat:added netcat acceptor -- (23, 1, 0, '', ('::', 6502, 0, 0))
2015-09-16 18:57:11,341 [6948]INFO:log:rbd.rproxy.OverloadedSite starting on 6501
2015-09-16 18:57:11,341 [6948]INFO:log:Starting factory <rbd.rproxy.OverloadedSite instance at 0x042CF9E0>
2015-09-16 18:57:11,342 [6948]INFO:log:rbd.rproxy.OverloadedSite starting on 6502
2015-09-16 18:57:11,342 [6948]INFO:log:Starting factory <rbd.rproxy.OverloadedSite instance at 0x042CFA58>
2015-09-16 18:57:11,345 [6904]INFO:dbsetup:last DB backup -- 2015-09-16
2015-09-16 18:57:11,345 [6904]DEBUG:dbsetup:installed DB connection <sqlite3.Connection object at 0x042D1320> for thread <Thread(Maintenance, started daemon 6904)>
2015-09-16 18:57:11,351 [6996]INFO:cacher:END validating cache contents
2015-09-16 18:57:11,354 [6504]INFO:rbd_watchdog_windows:starting Cacher
2015-09-16 18:57:11,355 [6504]INFO:cacher:BEGIN periodic cacher cleanup
2015-09-16 18:57:11,355 [6780]INFO:rbd_watchdog_windows:starting CGI
2015-09-16 18:57:11,355 [6504]DEBUG:dbsetup:installed DB connection <sqlite3.Connection object at 0x042D13A0> for thread <Thread(Cacher, started daemon 6504)>
2015-09-16 18:57:11,355 [6780]INFO:director:cgi server starting on port 9080
2015-09-16 18:57:11,355 [5596]INFO:rbd_watchdog_windows:starting RuleServer
2015-09-16 18:57:11,357 [7016]INFO:rbd_watchdog_windows:starting VC-Monitor
2015-09-16 18:57:11,358 [7056]INFO:rbd_watchdog_windows:starting Feedback
2015-09-16 18:57:11,358 [6504]DEBUG:cacher:flushed 0 item(s) from the cache
2015-09-16 18:57:11,358 [7016]DEBUG:dbsetup:installed DB connection <sqlite3.Connection object at 0x042D1420> for thread <Thread(VC-Monitor, started daemon 7016)>
2015-09-16 18:57:11,359 [7056]DEBUG:dbsetup:installed DB connection <sqlite3.Connection object at 0x042D1120> for thread <Thread(Feedback, started daemon 7056)>
2015-09-16 18:57:11,359 [6504]INFO:cacher:END periodic cacher cleanup
2015-09-16 18:57:11,361 [6780]INFO:director:removing old vib download dir -- c:\windows\temp\deploy.vib.bd_ksq
2015-09-16 18:57:11,361 [6780]INFO:director:using vib temp dir -- c:\windows\temp\deploy.vib.zarsrl
2015-09-16 18:57:11,362 [5596]DEBUG:dbsetup:installed DB connection <sqlite3.Connection object at 0x042D1520> for thread <Thread(RuleServer, started daemon 5596)>
2015-09-16 18:57:11,364 [7016]DEBUG:vc_servers:client SSL material -- C:\ProgramData\VMware\VMware vSphere Auto Deploy\ssl\rui.key, C:\ProgramData\VMware\VMware vSphere Auto Deploy\ssl\rui.crt
2015-09-16 18:57:11,364 [7056]DEBUG:vc_servers:client SSL material -- C:\ProgramData\VMware\VMware vSphere Auto Deploy\ssl\rui.key, C:\ProgramData\VMware\VMware vSphere Auto Deploy\ssl\rui.crt
2015-09-16 18:57:11,367 [6780]DEBUG:dbsetup:installed DB connection <sqlite3.Connection object at 0x042D17A0> for thread <Thread(CGI, started daemon 6780)>
2015-09-16 18:57:11,371 [5596]INFO:service_instance:rule server listening on port 9079
2015-09-16 18:57:12,293 [7016]WARNING:vc_monitor:Current VC(id: 1, attr: vc01.corp.demo.com, instance: 17233E11-13DC-4C08-AF6D-F159CA523962) currentTime : 2015-09-16 10:57:12.294488+00:00
2015-09-16 18:57:12,349 [7016]ERROR:vc_extension:Using https://192.168.50.13:6502 as the managementUrl
2015-09-16 18:57:12,371 [7016]INFO:vc_extension:registering servers with thumbprint -- EB:FC:D1:F7:C2:A7:0F:72:CE:B1:AF:4B:7F:77:0F:F9:70:6F:C4:4A
2015-09-16 18:57:12,384 [7016]INFO:vc_extension:updating rule extension info
2015-09-16 18:57:12,493 [6948]INFO:log:Starting factory <twisted.web.proxy.ProxyClientFactory instance at 0x04535648>
2015-09-16 18:57:12,494 [5596]DEBUG:pyvmomiserver:Connection from: ('127.0.0.1', 9088)
2015-09-16 18:57:12,496 [5596]DEBUG:pyvmomiserver:Queued workitem: (<socket._socketobject object at 0x044FD3B0>, ('127.0.0.1', 9088))
2015-09-16 18:57:12,496 [5976]DEBUG:pyvmomiserver:Handling workitem: (<socket._socketobject object at 0x044FD3B0>, ('127.0.0.1', 9088))
2015-09-16 18:57:12,496 [5976]DEBUG:pyvmomiserver:In do_POST: ('127.0.0.1', 9088)
2015-09-16 18:57:12,496 [5976]DEBUG:pyvmomiserver:<?xml version="1.0" encoding="UTF-8"?>
Hi
Please refer the KB
Hello
Please try below steps