Operational Defect Database
BugZero updated this defect 54 days ago.
Data sources
All data on this page is proprietary to BugZero® or gathered from public sources
VMware | 97117
Unable to access NSX-T UI and API failure following a change in Manager Node certificates
Last update date:
3/26/2024
Affected products:
NSX-T
Affected releases:
No affected releases provided.
Fixed releases:
No fixed releases provided.
Description:
Symptoms
You have recently changed the NSX Manager node or cluster certificatesYou are using third party certificates rather than the self signed NSX certificates which are created automatically when NSX is installed.API calls executed against the nodes fail The following entries are observed in /var/log/proxy/envoy.log [2024-03-22T14:24:52.685Z][22570][warning][config] [source/common/config/filesystem_subscription_impl.cc:43] Filesystem config update rejected: Error adding/updating listener(s) https-node-v4-local: Failed to load certificate chain from <inline>
Cause
There are multiple potential causes for this issue, but the most common are: The PEM document has been imported incorrectly rather than as per the below example: ----- BEGIN CERTIFICATE -----<leaf node certificate>----- END CERTIFICATE---------- BEGIN CERTIFICATE -----<intermediate ca certificate>----- END CERTIFICATE---------- BEGIN CERTIFICATE -----<root ca certificate>----- END CERTIFICATE----- The line separators in imported PEM file are DOS Style \r\n, which NSX does not correctly interpret. While NSX will allow the entry, the issue occurs when NSX eventually writes the PEM files for the for the associated service, they end with with \n\r\n between each line, which isn't correctly parsed.
Resolution
A fix for PEM files with DOS-style newlines was made in NSX-T 4.1.1.
Workaround
**It is recommended that backups are confirmed to be in place before making any changes**Step 1 - Re-import the CA-signed certificate. Instead of choosing "Browse" and selecting the certificate PEM file, open the PEM file in a text editor like Notepad and copy-paste the contents into the "Certificate Contents" field.Then, copy-paste the certificate's private key into the "Private Key" field. Be sure to un-select the "Service Certificate" option.Once the certificate has been imported, note its ID - that will be needed later and will be referred to as "ca-certificate-id"Log into any NSX manager node a root and run the following commands that will revert to the self-signed certificates created when the managers were installed. Step 2 - Determine the node IDs of the three manager nodes. These can be obtained from the System -> Appliances page in the NSX UI.For each manager node, open the "Details" link and click on the clipboard icon next to "UUID" in the details page. Note the UUIDs of these nodes.They will be referred to later as "nsx-mgr1-node_id", "nsx-mgr2-node_id", and "nsx-mgr3-node_id". Step 3 - Apply the new certificate to the manager nodes and VIP. Log into any manager node as root and run the following 4 commands: curl -X POST -H "x-nsx-username: admin" 'http://127.0.0.1:7440/nsxapi/api/v1/trust-management/certificates/<ca-certificate-id>?action=apply_certificate&service_type=API&node_id=<nsx-mgr1-node_id>'curl -X POST -H "x-nsx-username: admin" 'http://127.0.0.1:7440/nsxapi/api/v1/trust-management/certificates/<ca-certificate-id>?action=apply_certificate&service_type=API&node_id=<nsx-mgr2-node_id>'curl -X POST -H "x-nsx-username: admin" 'http://127.0.0.1:7440/nsxapi/api/v1/trust-management/certificates/<ca-certificate-id>?action=apply_certificate&service_type=API&node_id=<nsx-mgr3-node_id>'curl -X POST -H "x-nsx-username: admin" 'http://127.0.0.1:7440/nsxapi/api/v1/trust-management/certificates/<ca-certificate-id>?action=apply_certificate&service_type=MGMT_CLUSTER' Step 4 - Restart the reverse proxy service on each manager node. Log into each manager node as the admin user and run the comment "restart service http" Once the correct certificate has been applied, you can delete the bad certificate from the System -> Certificates page in the NSX Manager UI.
