Operational Defect Database
BugZero updated this defect 54 days ago.
Data sources
All data on this page is proprietary to BugZero® or gathered from public sources
VMware | 76359
Commission of a node in VMware Cloud Foundations 3.5.1 fails on "Configures DNS Server IPs on Given Hosts" task
Last update date:
3/26/2024
Affected products:
Cloud Foundation
Affected releases:
3.5
Fixed releases:
No fixed releases provided.
Description:
Symptoms
Commission of a node fails on "Configures DNS Server IPs on Given Hosts" task with error messages "HOST_DNS_CONFIGURATION_FAILED Cannot configure DNS on host".Similar errors messages as seen below are found in /var/log/vmware/vcf/operationsmanager/operationsmanager.log 2019-10-07 13:27:32.125 [vcf_om,c1151769a931b0d4,63498fe3ff22f45f] [-thread-11] DEBUG [c.v.v.hostmanager.comm.action.ConfigureDnsAction] <83b7cd40-e905-11e9-8177-e92dab19c32b> Nslookup command extract DNS Server: 150.80.1.5 server2019-10-07 13:27:32.125 [vcf_om,c1151769a931b0d4,63498fe3ff22f45f] [-thread-11] ERROR [c.v.v.hostmanager.comm.action.ConfigureDnsAction] <83b7cd40-e905-11e9-8177-e92dab19c32b> nslookup output validation failed. DNS [150.80.1.5, 150.80.1.6] configuration failed on the host 10.10.13.20.2019-10-07 13:27:32.127 [vcf_om,c1151769a931b0d4,63498fe3ff22f45f] [-thread-11] DEBUG [ com.vmware.evo.sddc.common.util.SshUtil] <83b7cd40-e905-11e9-8177-e92dab19c32b> Destroyed the ssh Session2019-10-07 13:27:32.140 [vcf_om,c1151769a931b0d4,63498fe3ff22f45f] [-thread-11] ERROR [c.v.e.sddc.orchestrator.model.error.ErrorFactory] [DLBFK7] HOST_DNS_CONFIGURATION_FAILED Cannot configure DNS [150.80.1.5, 150.80.1.6] on host 10.10.13.20 The SDDC manager in not able to parse the output of the '"nslookup " command and we are getting error messages "nslookup command output is empty" 2019-10-07 13:22:57.557 [vcf_om,c1151769a931b0d4,63498fe3ff22f45f] [-thread-11] DEBUG [ com.vmware.evo.sddc.common.util.SshUtil] <83b7cd40-e905-11e9-8177-e92dab19c32b> Starting to execute command [nslookup 10.10.13.20 ]2019-10-07 13:22:57.609 [vcf_om,c1151769a931b0d4,63498fe3ff22f45f] [-thread-11] DEBUG [ com.vmware.evo.sddc.common.util.SshUtil] <83b7cd40-e905-11e9-8177-e92dab19c32b> End of execution of command [nslookup 10.10.13.20], Status: 0Output:Error output:Command timed out: false2019-10-07 13:22:57.609 [vcf_om,c1151769a931b0d4,63498fe3ff22f45f] [-thread-11] ERROR [c.v.v.hostmanager.comm.action.ConfigureDnsAction] <83b7cd40-e905-11e9-8177-e92dab19c32b> nslookup command output is empty. DNS [150.80.1.5, 150.80.1.6] configuration failed on the host 10.10.13.20.2019-10-07 13:22:57.609 [vcf_om,c1151769a931b0d4,63498fe3ff22f45f] [-thread-11] DEBUG [c.v.v.hostmanager.comm.action.ConfigureDnsAction] <83b7cd40-e905-11e9-8177-e92dab19c32b> DNS configuration postvalidation:wait for 30000 miliseconds before retrying The Nslookup of the host FQDN and the Reverslookup of the host IP address from the SDDC manager and the host itself completed successfully.The host is configured with the correct DNS serverer [root@host1:~] cat /etc/resolv.confnameserver 150.80.1.5nameserver 150.80.1.6 Note: The preceding log excerpts are only examples. Date, time, and environmental variables may vary depending on your environment.
Cause
This is a known issue affecting VMware Cloud Foundation 3.5.x and the fix is to upgrade to VCF 3.7.0.
Resolution
This issue is resolved VMware Cloud Foundation version 3.7.0.
Workaround
Use the following steps to apply a patch in case an upgrade to VCF 3.7.0 is not an option and only if the current VCF deployment is on version 3.5.1. Download the operationsmanager_3.5.1-14878133.zip file attache to the KB.Use a file transfer utility to copy the operationsmanager_3.5.1-14878133.zip file to the /tmp directory on the SDDC Manager VM.SSH to the SDDC Manager Controller VM as the vcf user.Issue the following command to switch to the root user account: # su Extract the contents of the operationsmanager_3.5.1-14878133.zip file by issuing the following command: # unzip -d /tmp /tmp/operationsmanager_3.5.1-14878133.zip Issue the following command to stop the Operations Manager service: # systemctl stop operationsmanager Make a copy of the /opt/vmware/vcf/operationsmanager/vcf-operations-manager.jar file by issuing the following command: # cp -p /opt/vmware/vcf/operationsmanager/vcf-operations-manager.jar /opt/vmware/vcf/operationsmanager/vcf-operations-manager.jar.bak$(date +%Y%m%d) Replace the original /opt/vmware/vcf/operationsmanager/vcf-operations-manager.jar file with the version extracted in Step 4 by issuing the following command: # cp /tmp/operationsmanager_3.5.1-14878133/vcf-operations-manager.jar /opt/vmware/vcf/operationsmanager/ Change the ownership and permissions on the /opt/vmware/vcf/operationsmanager/vcf-operations-manager.jar file by issuing the following commands: # chown vcf_operationsmanager:vcf /opt/vmware/vcf/operationsmanager/vcf-operations-manager.jar# chmod 744 /opt/vmware/vcf/operationsmanager/vcf-operations-manager.jar Issue the following command to start the Operations Manager service: # systemctl start operationsmanager Note: It may take two to three minutes for the Operations Manager service to be functional after starting. Issue the following command to validate the state of the Operations Manager service. The Operations Manager status should be ACTIVE and the version is updated to "3.5.1-RELEASE-14878133": # curl http://localhost/operationsmanager/about | json_pp { "description" : "Operations Manager", "version" : "3.5.1-RELEASE-14878133", "status" : "ACTIVE", "serviceUrl" : "http://127.0.0.1/operationsmanager", "name" : "OPERATIONS_MANAGER", "id" : "3654b774-9ba0-4bc2-970f-cfd94ed84c43"} After applying the patch, the following steps must be completed prior to upgrading to a new VMware Cloud Foundation version: Log in to the SDDC Manager VM as the vcf user and then switch to the root user by typing su -.Issue the following command to make a copy of the /opt/vmware/vcf/lcm/lcm-app/conf/VersionAlias.yml file: # cp -fp /opt/vmware/vcf/lcm/lcm-app/conf/VersionAlias.yml /opt/vmware/vcf/lcm/lcm-app/conf/VersionAlias.yml.$(date +%Y%m%d) Open the /opt/vmware/vcf/lcm/lcm-app/conf/VersionAlias.yml file with a text editor and add the following to the end of the file. OPERATIONS_MANAGER: - base: 3.5.1-RELEASE-12050813 alias: - 3.5.1-RELEASE-14878133Note: Ensure that only spaces are used when editing VersionAlias.yml file as tab characters will break parsing and the LCM service may not start. Refer to commented examples in the file for a proper sample. Save and close the file.Issue the following command to restart the lcm service: # systemctl restart lcm Run the following command from SDDC Manager VM and make a note of the 'id'. # curl localhost/inventory/vcfservices | json_ppExample: { "status" : "ACTIVE", "description" : "Operations Manager", "name" : "OPERATIONS_MANAGER", "version" : "3.5.1-RELEASE-12050813", "serviceUrl" : "http://127.0.0.1/operationsmanager", "id" : "5eac87b3-c289-4dec-b1a7-d35a3bfe8e55" }, Run the following command to update the Operations Manager version value in the SDDC Manager inventory with the new version after applying the patch: # cqlsh --cqlversion=3.4.4 -e "update inventory.vcfservice set version='<OPERATIONS_MANAGER_Version>' where id='<OPERATIONS_MANAGER_ID>';" Notes: Replace this version with version value obtained from Step 6.Replace the 'id' with the 'id' value obtained step 6. Example: cqlsh --cqlversion=3.4.4 -e "update inventory.vcfservice set version='3.5.1-RELEASE-14878133' where id='5eac87b3-c289-4dec-b1a7-d35a3bfe8e55';" Verify the Operations Manager version was changed to “3.5.1-RELEASE-14878133" by running the command: curl localhost/inventory/vcfservices | json_pp’ Example: { "id" : "5eac87b3-c289-4dec-b1a7-d35a3bfe8e55", "status" : "ACTIVE", "name" : "OPERATIONS_MANAGER", "description" : "Operations Manager", "version" : "3.5.1-RELEASE-14878133", "serviceUrl" : "http://127.0.0.1/operationsmanager" }
