Operational Defect Database
BugZero updated this defect 52 days ago.
Data sources
All data on this page is proprietary to BugZero® or gathered from public sources
VMware | 91152
During Workload Management install, cluster selection returns "api request to NSX manager failed. Status code 403"
Last update date:
3/29/2024
Affected products:
vSphere
Affected releases:
7.08.0
Fixed releases:
No fixed releases provided.
Description:
Symptoms
When enabling Workload Management on vCenter connected to an NSX manager, after starting the configuration wizard and selecting NSX as the networking stack, there are no Clusters listed as COMPATIBLE.Clicking on the INCOMPATIBLE list, user will see the expected cluster listed as INCOMPATIBLE. Incompatibility Reasons will show: "Cluster domain-cX is missing compatible NSX-T VDS""Failed to list all distributed switches in vCenter <vCenter_ID>""API request to NSX Manager failed. Status code: 403, Status: 403" /var/log/vmware/wcp/wcpsvc.log on vCenter will report errors like: 2023-01-10T05:41:57.233Z debug wcp [opID=60dcb77a] nsx InitCompatibleClusterSet incompatibleReasons: [{vcenter.wcp.nsx.list.dvs.error Failed to list all distributed switches in vCenter 7785d3db-5e3e-4110-8997-7b35f97084ac. [7785d3db-5e3e-4110-8997-7b35f97084ac] map[] 0xc000b9bca0} {vcenter.wcp.nsx.manager.http.error API request to NSX Manager failed. Status code: 403, Status: 403 . [403 403 ] map[] 0xc000b9bb90}] /var/log/proton/nsxapi.log on NSX Manager reports errors like: 2023-01-10T05:41:57.233Z ERROR http-nio-127.0.0.1-7440-exec-6 AuthenticationEntryPoint 4768 - [nsx@6876 comp="nsx-manager" errorCode="MP403" level="ERROR" subcomp="http"] Not authenticated Full authentication is required to access this resource org.springframework.security.authentication.InsufficientAuthenticationException: Full authentication is required to access this resource at org.springframework.security.web.access.ExceptionTranslationFilter.handleAccessDeniedException(ExceptionTranslationFilter.java:194) ~[?:?] at org.springframework.security.web.access.ExceptionTranslationFilter.handleSpringSecurityException(ExceptionTranslationFilter.java:173) ~[?:?] /var/log/proxy/reverse-proxy.log on NSX Manager reports errors like: 2023-01-10T05:41:57.233Z INFO https-<VC_IP>-443-exec-9 VcTokenServices 5173 - [nsx@6876 comp="nsx-manager" level="INFO" subcomp="http"] The access token has been successfully validated locally.2023-01-10T05:41:57.233Z WARN https-<VC_IP>-443-exec-26 VcTokenServices 5173 - [nsx@6876 comp="nsx-manager" level="WARNING" subcomp="http"] Failed to get roles from JWT claims.2023-01-10T05:41:57.233Z ERROR https-<VC_IP>-443-exec-26 VcAuthTokenInterceptingFilter 5173 - [nsx@6876 comp="nsx-manager" errorCode="MP403" level="ERROR" subcomp="http"] Failed to Authenticate: User is not authorized to perform this operation on the application. Please contact the system administrator to get access. Using DCLI, the following errors are reported: # dcli +idcli> vcenter namespacemanagement distributedswitchcompatibility list --cluster domain-c8 --compatible falseServer error: com.vmware.vapi.std.errors.ErrorError messages: API request to NSX Manager failed. Status code: 403, Status: 403 . Failed to list all distributed switches in vCenter 7785d3db-5e3e-4110-8997-7b35f97084ac.
Cause
When vCenter connects to NSX-T manager in order to create Workload Management Supervisor Clusters, it delivers a JWT token to NSX to identify the user and privileges that will be used for object creation/management. This JWT token must contain the user ("wcp-<vCenterID>" user) and the roles the user can act as ("NsxAdministrators" in this case) in order for NSX to successfully return the requested information. If the user or privileges delivered in the JWT token are incorrect, NSX will return a 403 status code which indicates an authorization issue on the Client side (vCenter side). This problem can be caused by the following conditions: vCenter is not added as Compute Manager in NSX-T or is Down/Unregistered. NSX Administrators group in vSphere is not attached to the NsxAdministrator role. This group should contain the "wcp-<vCenterID>" user (user will look like: "wcp-bd2c292a-8477-4fac-ac0e-88e68bc774ff"),Or, the "wcp-<vCenterID>" user is not added as a member of the NSX Administrators groupOr, the "NsxAdministator" role has been deleted,Or, the "wcp-<vCenterID>" Solution User has been added to a non-default group in vSphere with limited privileges.
Impact / Risks
This error will prevent WCP Workload Management cluster instantiation.
Resolution
1. Ensure the vCenter is added to NSX-T manager as a Compute Manager and that it is Up and Registered: Log into NSX Manager GUI, click on System -> Fabric -> Compute ManagersIf the vCenter on which WCP is being enabled isn't added, add it.If the vCenter on which WCP is being enabled is added but shows either Registration Status or Connection Status as Not Registered or Down, correct the problem.It is also possible that the connection shows Up and Registered, but the certificate has changed or there is a stale entry. We can try to manually update the credentials with the following steps:Click the Checkbox next to the Compute ManagerSelect EDITTo the right of "FQDN or IP Address", select EDITRe-enter the vCenter "administrator@vsphere.local" username and password, then SAVE. 2. Ensure the "NSX Administrator" role is present on vCenter, if not, recreate it: From vSphere Web Client, select Menu -> AdministrationUnder Access Control, select RolesSearch for NSX Administrator, if the role is not present, recreate it:Connect to vCenter via SSHCreate a /usr/lib/vmware-wcp/py-modules/roles.py file and import config using the following command (ensure you copy the entire text from cat to the last EOF line): # cat <<EOF >>/usr/lib/vmware-wcp/py-modules/roles.pyimport featureStatefrom roles_groups_users_utils import Authzif __name__ == "__main__": authz_patch = Authz()authz_patch.ensure_roles_exist('/usr/lib/vmware-wcp/roles.xml')EOF Set the Python path with the following command: # export PYTHONPATH="${PYTHONPATH}:/usr/lib/vmware/site-packages" Run the python script to recreate roles using the following command: # python /usr/lib/vmware-wcp/py-modules/roles.py Example Output: # python /usr/lib/vmware-wcp/py-modules/roles.py 2023-01-10T18:09:54.288Z Role id 1004, name WorkloadStorageManagement - already exists2023-01-10T18:09:54.293Z Role id 1005, name vSphereKubernetesManager - already exists2023-01-10T18:09:54.296Z Role id 1007, name SupervisorServiceCluster - already exists2023-01-10T18:09:54.299Z Role id 1008, name SupervisorServiceRootFolder - already exists2023-01-10T18:09:54.302Z Role id 1009, name SupervisorServiceGlobal - already exists2023-01-10T18:09:54.306Z Role id 1021, name VMOperatorController - already exists2023-01-10T18:09:54.308Z Role id 1022, name VMOperatorControllerGlobal - already exists2023-01-10T18:09:54.311Z Role id 0, name VMServicesAdministrator - already exists2023-01-10T18:09:54.314Z Role id 0, name NsxAuditor - already exists2023-01-10T18:09:54.316Z Role id 0, name NsxViAdministrator - already exists2023-01-10T18:09:54.325Z Role id 0, name NsxAdministrator - added 3. Identify if the NsxAdministrators group exists and which users are added to it: # /usr/lib/vmware-vmafd/bin/dir-cli group list --name NsxAdministratorsEnter password for administrator@vsphere.local:CN=wcp-bd2c292a-8477-4fac-ac0e-88e68bc774ff,CN=ServicePrincipals,DC=vsphere,DC=local 4. Create group if needed. # /usr/lib/vmware-vmafd/bin/dir-cli ssogroup create --name NsxAdministrators 5. Add user to group if needed: First, gather the "wcp-<vCenterID>" username we will add to the "NsxAdministrators" group: # /usr/lib/vmware-vmafd/bin/dir-cli service listExample Output: # /usr/lib/vmware-vmafd/bin/dir-cli service list Enter password for administrator@vsphere.local:1. machine-bd2c292a-8477-4fac-ac0e-88e68bc774ff2. vsphere-webclient-bd2c292a-8477-4fac-ac0e-88e68bc774ff3. vpxd-bd2c292a-8477-4fac-ac0e-88e68bc774ff4. vpxd-extension-bd2c292a-8477-4fac-ac0e-88e68bc774ff5. hvc-bd2c292a-8477-4fac-ac0e-88e68bc774ff6. wcp-bd2c292a-8477-4fac-ac0e-88e68bc774ff Next, add the "wcp-<vCenterID>" user to the NsxAdministrators group using the following command: # /usr/lib/vmware-vmafd/bin/dir-cli group modify --name NsxAdministrators --add <solution_user_name>Example Command: # /usr/lib/vmware-vmafd/bin/dir-cli group modify --name NsxAdministrators --add wcp-bd2c292a-8477-4fac-ac0e-88e68bc774ff Example Output: # /usr/lib/vmware-vmafd/bin/dir-cli group modify --name NsxAdministrators --add wcp-bd2c292a-8477-4fac-ac0e-88e68bc774ff Enter password for administrator@vsphere.local:Account [wcp-bd2c292a-8477-4fac-ac0e-88e68bc774ff] added to group [NsxAdministrators]Group member [wcp-bd2c292a-8477-4fac-ac0e-88e68bc774ff] added successfully 6. Ensure the "NsxAdministrators" group is added to the "NSX Administrator" Role in vSphere Web Client: From vSphere Web Client, select Menu -> AdministrationUnder Access Control, select Global PermissionsSearch for VSPHERE.LOCAL\NsxAdministrators, ensure that it has role NSX Administrator assigned. If you don't see the user present, add it: Select ADDIn "Domain" field, select vsphere.local (or the local vmdir domain)In the "User/Group" field, select "NsxAdministrators" - ENSURE THERE ARE NO SPACES BEFORE OR AFTER THE NAMEIn the "Role" field, select "NSX Administrator"Check the "Propagate to childres" option, then click OK. 7. Ensure the wcp-<vCenterID> user is not added to any groups it shouldn't be added to: User should be added to only the following groups: SolutionUsersActAsUsersNsxAdministrators
