Operational Defect Database
BugZero updated this defect 44 days ago.
Data sources
All data on this page is proprietary to BugZero® or gathered from public sources
VMware | 79068
Partitioned vSAN cluster after upgrading vCenter Server on stretch cluster vSAN to version 6.6 or later
Last update date:
4/5/2024
Affected products:
vSAN
Affected releases:
6.x
Fixed releases:
No fixed releases provided.
Description:
Symptoms
The vSAN cluster became partitioned after upgrading.The pre-upgrade version did not support unicast.The upgraded version supports unicast.The vCenter Server VM is on the same vSAN it is managing.
Cause
This issue occurs while vCenter Server is pushing the updated unicast config (with "Supports Unicast" = true) to all the upgraded nodes. Since the witness node is the last node to be upgraded, its unicast config on all data nodes will be updated last (Example, flipping the unicast support for witness node from 'False' to 'True' on all data nodes). As these witness unicast config updates are happening, a partition may occur in one of two ways in the software upgrade process. Case 1 vCenter Server is upgraded to a release which supports unicast prior to 6.7u3, and ESXi is upgraded to any compatible release which supports unicast. After the witness node has been upgraded, vCenter Server updates the witness unicast config on all data nodes by removing the old config and adding the new config. After being told to remove the old witness unicast config, a node with all other updated unicast configs will leave the existing multicast cluster, will transition to the unicast channel and will join the new unicast cluster formed by the other nodes which had left the multicast cluster earlier.This sequential transition of nodes from the existing multicast cluster to the new unicast cluster can cause vCenter Server disk objects to become inaccessible preventing unicast config updates from completing and causing a partition.This issue can only happen on a vCenter Server deployed on the same vSAN setup. To identify if this issue has occurred, below are the sample unicast configs from two different nodes participating in these two different clusters. This node is running in unicast mode, all listed unicast agents have Supports unicast to be true, and witness unicast agent entry is missing.[root@ESX1:~] esxcli vsan cluster getCluster Information Enabled: true Current Local Time: 2020-03-10T22:22:24Z Local Node UUID: 5b04dac0-822d-0dee-a35a-506b4b3b7301 Local Node Type: NORMAL Local Node State: AGENT Local Node Health State: HEALTHY Sub-Cluster Master UUID: 5b04ed4a-7206-b308-0379-506b4b1adf01 Sub-Cluster Backup UUID: 5b083773-9885-b33c-e768-506b4b3b7b02 Sub-Cluster UUID: 52ab4afa-f4e5-f7a4-aac0-73188846b401 Sub-Cluster Membership Entry Revision: 11 Sub-Cluster Member Count: 4 Sub-Cluster Member UUIDs: 5b04dac0-822d-0dee-a35a-506b4b3b7301,5b083773-9885-b33c-e768-506b4b3b7b02,5b04c9e9-3833-ca3e-7f3c-506b4b3b7b03,5b04d090-f426-310e-d5b1-506b4b3b7f04 Sub-Cluster Member HostNames: ...... Sub-Cluster Membership UUID: 53fb675e-886e-7835-959e-506b4b1adf01 Unicast Mode Enabled: true Maintenance Mode State: OFF Config Generation: 278d30db-ce85-41d2-80a4-0205b9dd1d01 3 2020-03-10T20:40:36.357 [root@ESX1:~] esxcli vsan cluster unicastagent listNodeUuid IsWitness Supports Unicast IP Address Port Iface Name Cert Thumbprint------------------------------------ --------- ---------------- ------------- ----- ---------- -----------------------------------------------------------5b04dac0-822d-0dee-a35a-506b4b3b7301 0 true 10.10.16.1 12321 ......5b083773-9885-b33c-e768-506b4b3b7b02 0 true 10.10.16.2 12321 ......5b04c9e9-3833-ca3e-7f3c-506b4b3b7b03 0 true 10.10.16.3 12321 ......5b04d090-f426-310e-d5b1-506b4b3b7f04 0 true 10.10.16.4 12321 ......Below is from the host in the multicast cluster partition. See that this node is running in multicast mode, and witness unicast agent entry have Supports Unicast as false. [root@ESX5:~] esxcli vsan cluster getCluster Information Enabled: true Current Local Time: 2020-03-10T22:23:24Z Local Node UUID: 5b04dac0-822d-0dee-a35a-506b4b3b7305 Local Node Type: NORMAL Local Node State: AGENT Local Node Health State: HEALTHY Sub-Cluster Master UUID: 5b04ed4a-7206-b308-0379-506b4b1adf05 Sub-Cluster Backup UUID: 5b083773-9885-b33c-e768-506b4b3b7b06 Sub-Cluster UUID: 52ab4afa-f4e5-f7a4-aac0-73188846b401 Sub-Cluster Membership Entry Revision: 11 Sub-Cluster Member Count: 4 Sub-Cluster Member UUIDs: 5b04dac0-822d-0dee-a35a-506b4b3b7305,5b083773-9885-b33c-e768-506b4b3b7b06,5b04c9e9-3833-ca3e-7f3c-506b4b3b7b07,5b04d090-f426-310e-d5b1-506b4b3b7f08 Sub-Cluster Member HostNames: ...... Sub-Cluster Membership UUID: 53fb675e-886e-7835-959e-506b4b1adf02 Unicast Mode Enabled: true Maintenance Mode State: OFF Config Generation: 278d30db-ce85-41d2-80a4-0205b9dd1d01 3 2020-03-10T20:40:36.357 [root@ESX5:~] esxcli vsan cluster unicastagent listNodeUuid IsWitness Supports Unicast IP Address Port Iface Name Cert Thumbprint------------------------------------ --------- ---------------- ------------- ----- ---------- -----------------------------------------------------------00000000-0000-0000-0000-000000000000 1 false 10.10.20.1 123215b04dac0-822d-0dee-a35a-506b4b3b7305 0 true 10.10.16.5 12321 ......5b083773-9885-b33c-e768-506b4b3b7b06 0 true 10.10.16.6 12321 ......5b04c9e9-3833-ca3e-7f3c-506b4b3b7b07 0 true 10.10.16.7 12321 ......5b04d090-f426-310e-d5b1-506b4b3b7f08 0 true 10.10.16.8 12321 ...... Case 2 vCenter Server is upgraded to a release 6.7u3 or later, and ESXi is upgraded to any release that supports unicast.The software upgrade sequence remains same as discussed in Case 1, except that for the last step which updates all data nodes with the new witness unicast config. The vCenter Server on or after 6.7u3 does not update data nodes with a new witness unicast config after the witness node upgrades. So, the cluster does not get partitioned at this point and keeps using multicast channel for network communication. The actual witness unicast config update takes place when a cluster remediation event happens. For example, this remediation trigger could be a forced remediation through health check UI, or a RemediateVsanCluster API invocation through a script, or DFC, etc.In the initial phase of cluster remediation, the vCenter Server will first try to update the witness unicast config on all data nodes as seen in Case 1. As a result, vCenter Server will update the witness unicast config on all data nodes. This might again lead to same issue and is subjected to the similar conditions/diagnoses.
Impact / Risks
If software upgrade causes a cluster to be partitioned, manual intervention is needed to fix the witness unicast configuration on all data nodes.Some VMs (including the vCenter Server VM) may become inaccessible and their data may not be available.
Resolution
Healing a partitioned cluster In case a partition occurs, it can be healed through either a series of manual steps or a script that runs in parallel on all hosts. Manually updating the cluster To resolve this issue the Supports Unicast=false will need to manually be updated to true.Note: The automated approach explained below is highly recommended instead of the manual approach here.Note: These steps need to be completed on each host. List the unicast config with this command: esxcli vsan cluster unicastagent list NodeUuid IsWitness Supports Unicast IP Address Port Iface Name Cert Thumbprint------------------------------------ --------- ---------------- ------------- ----- ---------- -----------------------------------------------------------00000000-0000-0000-0000-000000000000 1 false 10.10.20.1 123215b04dac0-822d-0dee-a35a-506b4b3b7305 0 true 10.10.16.5 12321 ......5b083773-9885-b33c-e768-506b4b3b7b06 0 true 10.10.16.6 12321 ......5b04c9e9-3833-ca3e-7f3c-506b4b3b7b07 0 true 10.10.16.7 12321 ......5b04d090-f426-310e-d5b1-506b4b3b7f08 0 true 10.10.16.8 12321 ...... Any unicast agent entry with Supports Unicast = false, will need to be removed: esxcli vsan cluster unicastagent remove -a xx.xx.xx.x For example: esxcli vsan cluster unicastagent remove -a 10.10.20.1 If the entry is missing Add the unicast entry manually:esxcli vsan cluster unicastagent add -a xxx.xxx.xxx.xxx -u NodeUUID -U 1 -t witness For example:esxcli vsan cluster unicastagent add -a 10.10.20.1 -u 5b04dac0-822d-0dee-a35a-506b4b3b7310 -U 1 -t witness Using the Script Download the attached remediate_witness.py to a machine that has access to all ESXi hosts and is not part of the vSAN cluster.Run this script with this command: python remediate_witness.py -h host1[,host2][,...] -u [user] -p [password] -a [WITNESS_IP] -U [WITNESS_UUID] [-D] Note: Python variables may require the path to run the script, for example:# python remediate_witness.py -h host1[,host2][,...] -u [user] -p [password] -a [WITNESS_IP] -U [WITNESS_UUID] [-D]This script needs python paramiko environment to run, different environment settings may be needed for this lib.Appending -D option (dryrun), will output a test run of the script. Example: # python remediate_witness.py -h 10.184.100.218,10.184.107.87,10.184.105.206,10.184.102.27 -a 1.2.3.4 -U 5eb379df-df1c-cb9d-cf70-02000422ea91DEBUG: Remediate hosts: ['10.184.100.218', '10.184.107.87', '10.184.105.206', '10.184.102.27']10.184.100.218: esxcli vsan cluster unicastagent remove -a 1.2.3.4; esxcli vsan cluster unicastagent add -a 1.2.3.4 -u 5eb379df-df1c-cb9d-cf70-02000422ea91 -U 1 -t witness10.184.107.87: esxcli vsan cluster unicastagent remove -a 1.2.3.4; esxcli vsan cluster unicastagent add -a 1.2.3.4 -u 5eb379df-df1c-cb9d-cf70-02000422ea91 -U 1 -t witness10.184.102.27: esxcli vsan cluster unicastagent remove -a 1.2.3.4; esxcli vsan cluster unicastagent add -a 1.2.3.4 -u 5eb379df-df1c-cb9d-cf70-02000422ea91 -U 1 -t witness10.184.105.206: esxcli vsan cluster unicastagent remove -a 1.2.3.4; esxcli vsan cluster unicastagent add -a 1.2.3.4 -u 5eb379df-df1c-cb9d-cf70-02000422ea91 -U 1 -t witnessDEBUG: Running command=esxcli vsan cluster unicastagent remove -a 1.2.3.4; esxcli vsan cluster unicastagent add -a 1.2.3.4 -u 5eb379df-df1c-cb9d-cf70-02000422ea91 -U 1 -t witness on host 10.184.100.218DEBUG: Running command=esxcli vsan cluster unicastagent remove -a 1.2.3.4; esxcli vsan cluster unicastagent add -a 1.2.3.4 -u 5eb379df-df1c-cb9d-cf70-02000422ea91 -U 1 -t witness on host 10.184.102.27DEBUG: Running command=esxcli vsan cluster unicastagent remove -a 1.2.3.4; esxcli vsan cluster unicastagent add -a 1.2.3.4 -u 5eb379df-df1c-cb9d-cf70-02000422ea91 -U 1 -t witness on host 10.184.105.206DEBUG: Running command=esxcli vsan cluster unicastagent remove -a 1.2.3.4; esxcli vsan cluster unicastagent add -a 1.2.3.4 -u 5eb379df-df1c-cb9d-cf70-02000422ea91 -U 1 -t witness on host 10.184.107.87DEBUG: Result rc=0 stdout= stderr= on host 10.184.107.87DEBUG: Result rc=0 stdout= stderr= on host 10.184.102.27DEBUG: Result rc=0 stdout= stderr= on host 10.184.105.206DEBUG: Result rc=0 stdout= stderr= on host 10.184.100.218 Preventing the cluster partitioning To avoid the partitioning from occurring, the vCenter Server must be on 6.7 U3 or later. As described above in Case 2, the upgrade of ESXi to 6.5/6.7 will not trigger the multicast to unicast change on the cluster (this can be confirmed with "esxcli vsan cluster get" to check on each host). So, the above script can be run proactively to fix the witness unicast config on all data nodes. This script executes these commands on all hosts at the same time in parallel, so that all hosts can be updated within 30 seconds to avoid the APD timeout (which is why manual steps are not recommended). This proactive step will make sure that there will be no partition during cluster remediation leading to the VC going inaccessible. Run this script as quickly as possible after the ESXi upgrade, and avoid making any configuration changes (see the below list) until script is done. Once this is done, one can check whether each host is in expected state by running esxcli vsan cluster get command (no partition and running in unicast mode).Note that the script will cause a temporary partition (due to hosts moving from the multicast to unicast mode) which will heal quickly within 30 seconds. However, if script runs into any issue which prevents it to fix a host's unicast config, it will cause a cluster partition which will not heal without the manual intervention. Below are some of the things which can be verified before running the script. Verify network connectivity between jump Host (Linux/Windows) which will run the script to all the hosts in the cluster.Verify that root password is the same for all the cluster hosts.Do the dry run first. Make sure script is run from jump host(Linux/Windows) which is not running on vSAN datastore. As described above in Case 2, witness unicast config update will take place automatically(we want to avoid this) if a cluster remediation event happens. Below is a list of changes and user actions which can indirectly cause a cluster remediation event. These changes and actions should be avoided after upgrading the ESXi hosts and before running the scripts. Note that this is not an exhaustive list. To avoid any possible change to the system after the upgrade, and run the script as soon as possible. Network configuration changeESXi configuration changeForce health checkRemediateVsanCluster API invocation through a scriptDFCManagement operation to the hosts and VC.
