Operational Defect Database
BugZero found this defect 4582 days ago.
Data sources
All data on this page is proprietary to BugZero® or gathered from public sources
Hewlett Packard Enterprise | c02720892
CUSTOMER ADVISORY: (REVISED) HP B-series 8 Gb switches or port blades running with FOS v6.4.x might observe a loss of connectivity on a port or trunked ISL causing all traffic to be dropped
Last update date:
2/28/2024
Affected products:
Brocade 8Gb SAN Switch for HPE BladeSystem c-Class
HPE 1606 Extension SAN Switch
HPE 4400 Enterprise Virtual Array
HPE 8/24 SAN Switch
HPE 8/40 SAN Switch
HPE 8/8 SAN Switch
HPE 8/80 SAN Switch
HPE Converged Network Switches
HPE Encryption SAN Switch
HPE Storage SAN Director Switch
HPE StorageWorks 4/256 SAN Director Switch
HPE B-series Encryption Option
Affected releases:
No affected releases provided.
Fixed releases:
No fixed releases provided.
Description:
Info
Revision Table Version Date Details 2 27–Oct–2011 Revised content regarding disabling a trunked group of ports in the Resolution section. 1 17–Feb–2011 Original release. The Fabric OS (FOS) manages a routing table within the ASIC to indicate both domain and area routes for external facing ports. An issue with FOS v6.4.0x causes incorrect programming of the routing tables where a duplicate-assigned routing entry is created. Disabling a port not only removes the routing associated with the disabled port, but also removes routing information associated with the duplicate port that is still online. All traffic is dropped on the duplicate-assigned port or trunk group with a duplicate-assigned routing table entry, while the other duplicate-assigned port is offline. If the disabled port is part of either an ISL trunk or an F_Port trunk, all members of that trunk lose routing information, and therefore drop frames. To create duplicate entries, which are the root cause of this issue, all of the following must occur: A set of trunked ports consisting of two or more trunked links must exist, and one or more offline or disabled ports must also exist on the same ASIC. If all other ports on the same ASIC as the trunked links are online , this issue will not be exposed. No duplicate assignments are possible. A firmware upgrade to an affected FOS v6.4.x from any other version of FOS or an HA Failover while running an affected FOS v6.4.x firmware must take place. The issue marks previously offline ports routine table entries as being available for assignment. Online ports are still handled correctly. The trunked ISL must go offline and then back online in a specific order. All links of the trunk must be disabled. If at least one link of the trunk remains online, even if the trunk master changes, no duplicate links will be created and the issue will be avoided. Once the last port of the trunk is disabled, any other link of the trunk must come online prior to the link that was the last one to go offline. The remaining links might come online in any order. After the above step, the newly-assigned trunk master might be given a duplicate routing table entry as one of the currently offline ports. If the last link to go offline is also the same link to come online, duplicates are not assigned and the issue is avoided. If a member of a trunk group is removed and replaced by a single F_Port or E_Port that does not rejoin the trunk group, the port might also be assigned a duplicate routing table entry. A new device is added, or an offline port is enabled and brought online. If the table entry of this port was assigned to the trunked group, the switch might encounter the failure. When either of the duplicate-assigned ports is disabled, the routing information is cleared, and the port that remained online will lose all routes because the same routing table offset is being used. This causes the port to stop passing traffic. If the online duplicate is a member of the trunked group, then all links of the trunk will fail to pass traffic. If the trunked group is taken offline and then brought back online with the first online link being different from the last link to go offline, the duplicate assignment might change to another offline port. If multiple-trunked groups exist on the same ASIC, then both of them might cause duplicate entries to be assigned.
Scope
NOTE: 4 Gb blades within an HP DC SAN Backbone Director Switch or DC04 SAN Director Switch are not affected. 8 Gb blades within an HP 4/256 SAN Director Switch director are vulnerable to this issue. This issue affects the platforms and blades running FOS v6.4.0a, v6.4.0b, v6.4.0c, v6.4.1. (The affected platforms are listed at the end of this document.)
Resolution
HP recommends upgrading to FOS 6.4.1a or later version. When upgrading to a firmware version that contains the fix, the firmware will attempt to automatically correct and re-assign any existing duplicate entries that might exist. Even if no symptoms have been observed, because the offline port has not been enabled, the firmware will detect and correct the duplicate entries where possible. When both of the duplicate ports are online at the time of upgrade, the firmware cannot automatically repair and re-assign table entries without causing a bounce of the affected ports. Instead, the firmware will indicate that duplicate table entries have been detected and were not corrected. The following RAS log messages are examples of what will be presented when duplicates remain after an upgrade: 2010/11/23-10:00:39, [C2-1013], 4687, SLOT 6 | CHASSIS, WARNING, DCX_1, S2,P24(16): Duplicate rte_tbl_select detected! 2010/11/23-10:00:39, [C2-1013], 4688, SLOT 6 | CHASSIS, WARNING, DCX_1, S2,P287(38): Duplicate rte_tbl_select detected! To fully repair and re-assign table entries, all ports that are identified as having a duplicate table select entry should be bounced (disabled and then enabled). In this example, ports 24 and 287 have been identified as not being repaired, and duplicate routing table entries exist. When a trunked group is identified as being a duplicate, only one of the ports within the group will be identified. The port identified might or might not be the trunk master. If any port within a trunked group is identified, then the entire trunk group will need to be bounced. When disabling a trunked group of ports, disable all slave ports followed by the trunk master. Once all ports of the trunk are disabled, enable the ports one at a time. Order does not matter. If the RAS log message points to an internal port, those messages may be ignored. 2011/08/27-18:26:10, [C2-1013], 100, SLOT 7 | CHASSIS, WARNING, DCX_01, S1,P-1(42): Duplicate rte_tbl_select detected! The table values are not used on internal ports and will not affect the routes of front-end ports. The C2-1013 message will only be displayed after a hafailover/hareboot event of the CPs/switch. To clear the condition of these messages being seen further on back-end ports, the blade/switch must be power cycled. NOTE: Due to a recently identified encryption issue, HP does not recommend upgrading the HP Encryption SAN Switch or DC Backbone Switch Encryption Fibre Channel Blade Option to v6.4.1a. Instead, HP recommends upgrading to v6.4.1b or later in order to fix both issues. Workaround A switch reboot clears the duplicate assignments and restores all ASIC tables to a proper state. In an HP DC SAN Backbone Director Switch, DC04 SAN Director or 4/256 SAN Director Switch with 8 Gb blades, a slot power off/slot power on operation restores all ASICs on that one blade to the proper state. However, the condition might return if an HA Failover or firmware upgrade to version 6.4.x that does not contain the fix occurs. Enabling both of the duplicate-assigned ports, and keeping both of these ports online will avoid any loss of traffic until the fix can be applied. Disabling or enabling the trunked group to bring all ports offline and back online to move the duplicate assignment to another unused or offline port might also work. This action is not guaranteed to succeed as a workaround. This repair might appear to be successful; however, the duplicate routing table might match another offline port, which could result in the issue recurring. Proactive Updates Receive support alerts (such as Customer Advisories), driver updates, software, firmware, and customer replaceable components, in your e-mail through HP Subscriber's Choice. Sign up for Subscriber's Choice Driver, Patch, Security, and Support alerts at the following URL: http://www.hp.com/go/myadvisory
