Operational Defect Database
BugZero updated this defect 54 days ago.
Data sources
All data on this page is proprietary to BugZero® or gathered from public sources
VMware | 97115
Cloud Director UI unavailable after replacing the Cloud Director Appliance Management Certificates
Last update date:
3/26/2024
Affected products:
Cloud Director
Affected releases:
10.4
Fixed releases:
No fixed releases provided.
Description:
Symptoms
Cloud Director UI could not be reached after following the Cloud Director documentation to Replace the Appliance Management Key-Certificate Pair.The new Appliance Management certificate is signed by an internal certificate authority which would not be recognised by default.Cloud Director /opt/vmware/vcloud-director/logs/vcloud-container-debug.log files show an error connecting to the database similar to: | ERROR | processor-ContentLibrary | JDBCExceptionReporter | SSL error: org.springframework.transaction.CannotCreateTransactionException: Could not open Hibernate Session for transaction; nested exception is org.hibernate.exception.JDBCConnectionException: Cannot open connection | PostgreSQL /var/vmware/vpostgres/current/pgdata/log/postgresql-<DATE>.log files on the Primary Cell show client connection errors of the form: LOG: could not accept SSL connection: sslv3 alert certificate unknown Appliance Sync service /opt/vmware/var/log/vcd/appliance-sync.log files on all Cells show that their truststores have been updated with the new Appliance Management certificate: | Updating VCD trust store - processing directory node-<NODE_UUID> The /opt/vmware/vcloud-director/etc/truststore.pem on all Cells contains the new Appliance Management certificate.
Cause
This issue occurs if the Appliance Management certificate is signed by an internal certificate authority which would not be recognised by default.The Cloud Director service will only trust the certificate if it or the certificate authority certificate is present in its truststore during service startup.
Impact / Risks
Backup Cloud Director before making any changes, Backup and Restore of VMware Cloud Director Appliance. Stopping the Cloud Director services on the Cells will make Cloud Director unavailable so schedule maintenance for the change as appropriate.
Resolution
For Cloud Director 10.5.1 and later follow the updated steps which include stopping and restarting the Cloud Director services as part of the process, Replace or Renew the VMware Cloud Director Appliance Management Certificates. To resolve the issue in earlier versions of Cloud Director simply stop the Cloud Director services on the Cells before changing the Appliance Management certificate and start the Cloud Director services on the Cells after the new certificate has been applied. Example steps would be as follows: Before changing the certificate schedule a downtime and stop the Cloud Director service on all Cells in the cluster, the guest OS of the Cells does not need to be shutdown: /opt/vmware/vcloud-director/bin/cell-management-tool -u <VCD_ADMIN_USERNAME> cell --shutdown systemctl stop vmware-vcd Proceed to apply the new certificate to the Cells as per the Cloud Director documentation, Replace the Appliance Management Key-Certificate Pair.After replacing the certificate and key, and restarting the Appliance VAMI and PostgreSQL services wait 2 minutes to ensure the Appliance Sync service of all the Cloud Director Cells is able to update the truststores with the Cell's new cert. The Appliance Sync logs can be followed to confirm that this is occurring, it should update approximately every ~60 seconds: tail -f /opt/vmware/var/log/vcd/appliance-sync.log | grep "Executing vcd appliance sync scripts\|Updating VCD trust store\|Successfully completed run of appliance sync script" Confirm the new certificate is present in all the Cells' truststore: less -i /opt/vmware/vcloud-director/etc/truststore.pem Restart the Cloud Director service again on all the Cells: systemctl start vmware-vcd Confirm that the Cloud Director Provider and Tenant UIs become available once the services have finished startup.
