Operational Defect Database
BugZero updated this defect 46 days ago.
Data sources
All data on this page is proprietary to BugZero® or gathered from public sources
VMware | 88034
NSX-T Edge vMotion best practices
Last update date:
4/3/2024
Affected products:
NSX-T
Affected releases:
No affected releases provided.
Fixed releases:
No fixed releases provided.
Description:
Details
vMotion is an essential vSphere capability that facilitates the live migration of VMs between ESXi hosts.NSX-T Data Center fully supports the vMotion of Edge VMs since version 2.5.1.The NSX-T Edge is a critical network datapath component processing North South traffic and other NSX-T services such as Load Balancers, VPNs, NAT, etc.vMotion can expose underlying environmental issues that may result in minor disruption to VM network connectivity during migration.In a busy vSphere cluster configured with fully automated DRS, DRS will migrate VMs to balance resource usage across the ESXi hosts.However when making decisions to move VMs, DRS is not aware of whether a VM is a test VM burning CPU or a production critical Edge VM handling datacenter North South traffic.It is therefore recommended to reduce Edge VM vMotion to only migrations that are necessary e.g ESXi maintenance mode etc. Note: If Edges are used for L2 Bridging, this function is not vMotion aware. The physical switches will not know of the bridged MACs move to another ESXi host until it sees outbound traffic sourced from these MAC addresses. This can result in blackholing of traffic. Therefore, if vMotion of an active Bridge Edge is required, VMware recommends first to failover the Bridge to the standby Edge and then vmotion the former Active Edge node.Note: Storage vMotion is supported but it is recommended to put the Edge in NSX maintenance mode prior to performing this activity to ensure it is non disruptive.
Solution
DRS rules can be used to protect Edges VMs from unnecessary vMotion activity.First a DRS VM group should be created for each Edge VM and a corresponding DRS ESX host group for the host the VM will run on.Ensure VLAN/MTU config on TORs where the Edge may be vMotioned is consistent. Define the DRS vm/host rules such that the underlying hosts for the edge VM are on the same L2 domain and that the BGP/OSPF neighbors or static route next hops defined on Tier0 are reachable from all the hosts where the Edge may be vmotioned.If more than one Edge VM is required to run on a host then add multiple Edge VMs to the VM group instead of just 1.1. Browse to the cluster in the vSphere Client.2. Click Configure.3. Under Configuration, select VM/Host Groups and click Add.4. For each Edge VM add a VM Group a. In the Create VM/Host Group dialog box, type an appropriate name for the group e.g Edge1 b. Select VM Group from the Type drop down box and click Add. c. Click the check box next to the Edge VM and add it. d. Click OK.5. For each ESXi host add a Host Group a. In the Create VM/Host Group dialog box, type an appropriate name for the group e.g ESX1 b. Select Host Group from the Type drop down box and click Add. c. Click the check box next to the ESX host and add it. d. Click OK.Next a rule will be created to map the VM groups to the Host groups1. Browse to the cluster in the vSphere Client.2. Click Configure.3. Under Configuration, select VM/Host Rules and click Add.4. In the Create VM/Host Rule dialog box, type a name for the rule e.g. Edge1-ESX1.5. From the Type menu, select Virtual Machines to Hosts.6. From the drop downs match the VM Group e.g. Edge1 with the Host Group e.g. ESX1 where is should run.7. Select Should run on hosts in group.8. Click OK.These rules will ensure that when DRS is making decisions to move VMs it will try to keep the Edge VM running on the mapped host when possible.If that ESX host is entered into maintenance mode, the Edge VM will be moved automatically and afterwards when the host exits maintenance mode it will be moved back automatically.These rules also serve the purpose of ensuring that Edges in the same Edge cluster run on separate ESX hosts providing redundancy from a host failure standpoint.
