Operational Defect Database
BugZero found this defect 448 days ago.
Data sources
All data on this page is proprietary to BugZero® or gathered from public sources
Hewlett Packard Enterprise | a00130365en_us
Advisory: HPE B-series Switches - Gen7 Platforms May Encounter a Timing Condition That Causes an Internal Memory Buffer to Falsely Determine it is Always Full
Last update date:
2/28/2024
Affected products:
HPE Storage Fibre Channel Switch B-series SN6700B
HPE Storage Fibre Channel Switch B-series SN6750B
HPE Storage SAN Director Switch
Affected releases:
No affected releases provided.
Fixed releases:
No fixed releases provided.
Description:
Info
Gen7 HPE SN8700B Fibre Channel Director Switch, HPE SN6750B Fibre Channel Switch and HPE SN6700B Fibre Channel Switch running Fabric OS (FOS) v9.1.1, v9.1.1_01 or v9.1.1a can encounter a timing condition that causes an internal memory buffer to falsely determine that it is always full. If encountered, this can cause excessive compact flash writes due to FOS recording repeated “buffer full” internal RASlog messages. A pre-allocated memory buffer within FOS, shared between all ASICs for communication up to the CPU, may potentially encounter a timing condition, which can cause buffer entries to be lost when they are freed. Over time, if this internal buffer loses all of its entries through repeated instances of this timing window, then the buffer will always appear full whenever any ASIC attempts to communicate up to the CPU through this buffer. Every attempted write to this buffer, while it is “full,” fails, and an internal RASlog message is generated indicating that a “buffer full” condition was encountered. However, as this buffer always appears to be full, this same RASlog message is generated on every attempted write. The repeated RASlogs are written to the compact Flash with every attempt to write into the buffer. Over a prolonged period, these compact flash write operations could have an impact on the compact flash. If left unaddressed, over a period of months, the excessive writes could cause the compact flash to fail. To encounter the timing condition, prolonged periods of excessive SCSI-control frames within a single second are also required. This is typically caused by servers and storage platforms that utilize a repeated use of “reserve,” “release,” or “test unit ready” commands. Excessive use of “aborts” or “inquiry” commands can also lead to the timing condition. As the RASlogs are internal messages, used later by support for triage, there may be no visible symptoms. The internal RASlog can be viewed from the “Maintenance” account, which is typically used by support teams when troubleshooting issues. From the maintenance account, the following command can be used to view the internally generated RASlogs: maintenance> serviceexec "errdumpall" If repeated patterns of the following RASlogs are observed, then this system is currently experiencing the interpreted out-of-buffer entry failure condition: C5-9001 and C5-5064 OR GE5-9001 and GE5-5064
Scope
Only Gen7 switches (HPE SN8700B FC Director Switch, HPE SN6750B FC Switch, HPE SN6700B FC Switch) are at risk of encountering this issue. Additionally, the timing condition is only exposed within the FOS v9.1.1, v9.1.1_01 and v9.1.1a firmware versions. FOS versions prior to v9.1.1 are not exposed, including v9.1.0b and all v9.0.1x Target Path versions.
Resolution
A software solution provided in FOS v9.1.1b prevents any loss of entries from the internal buffers, effectively preventing the condition where excessive compact flash writes could occur due to ASIC communication used by flow vision. Any system that is currently encountering the condition and excessive compact flash writes are automatically corrected upon upgrade to v9.1.1b or later firmware levels. No additional actions beyond upgrading the Fabric OS is required. HPE recommends that all Gen7 platforms (HPE SN8700B FC Director Switch, HPE SN6750B FC Switch, HPE SN6700B FC Switch) should immediately be upgraded to FOS v9.1.1b (or later versions) to prevent exposure to the failure condition that could cause excessive compact flash writes, as the condition is not visible to the user, and over time could cause the compact flash to fail. As a workaround, if a switch encounters excess compact flash writes (repeating patterns of C5-9001/C5-5064 or GE5-9001/GE5-5064 RASlogs) then temporarily stop use of the internal buffer by deactivating flow vision monitoring on the switch using the following command: admin> flow –deactivate sys_flow_monitor Re-activate flow vision monitoring after upgrading to a FOS version that contains a solution to prevent the loss of internal buffer entries using the following command: admin> flow –activate sys_flow_monitor IMPORTANT : These commands must be performed on each virtual fabric (VF) separately to fully stop use of the buffer. RECEIVE PROACTIVE UPDATES : Receive support alerts (such as Customer Advisories), as well as updates on drivers, software, firmware, and customer replaceable components, proactively in your e-mail through HPE Support Alerts. Sign up for Support Alerts at the following URL: HPE Email Preference Center .
