Operational Defect Database
BugZero found this defect 494 days ago.
Data sources
All data on this page is proprietary to BugZero® or gathered from public sources
Hewlett Packard Enterprise | a00048622en_us
Advisory: (Revision) HPE Integrated Lights-Out 4 (iLO 4) – How to Format the NAND Used to Store AHS logs, OneView Profiles, and Intelligent Provisioning
Last update date:
3/15/2024
Affected products:
HPE ConvergedSystem 500 for SAP HANA Scale-out Configurations
HPE ConvergedSystem 500 for SAP HANA Scale-up Configurations
HPE ProLiant BL420c Gen8 Server Blade
HPE ProLiant BL460c Gen8 Server Blade
HPE ProLiant BL460c Gen9 Server Blade
HPE ProLiant BL465c Gen8 Server Blade
HPE ProLiant BL660c Gen8 Server Blade
HPE ProLiant DL120 Gen9 Server
HPE ProLiant DL180 Gen9 Server
HPE ProLiant DL320e Gen8 Server
HPE ProLiant DL320e Gen8 v2 Server
HPE ProLiant DL360 Gen9 Server
Affected releases:
No affected releases provided.
Fixed releases:
No fixed releases provided.
Description:
Info
Document Version Release Date Details 9 January 12, 2023 Updated utility link in NAND Format Method 3 8 January 12, 2022 Updated the download links/URLs for support center, iLO 4, and an advisory link 7 January 5, 2022 Updated the Linux command string to "./hponcfg -i Force_Format.xml 6 April 21. 2021 Updated HP Config Utility link in NAND Option 3 and iLO 4 download link 5 November 7, 2018 Added additional information in the Resolution section 4 September 19, 2018 Updated instances of iLO 4 firmware v2.60 to v2.61 3 August 1, 2018 Updated Resolution, additional formatting changes. 2 July 31, 2018 Updated Resolution, formatting changes and other minor changes. 1 June 6, 2018 Original Document Release. NAND corruption may require that the NAND be formatted.
Scope
Any HPE ProLiant Gen8-series or HPE ProLiant Gen9-series servers with iLO 4.
Resolution
Prerequisites for formatting the NAND The user must log into iLO 4 with "administrator" level credentials Many of the methods below require an .XML file (which needs to be created and named "Force_Format.xml"). The content of the .XML file needs to be similar to the following, with the iLO administrator username and password replaced for Administrator and Password on line 2: <RIBCL VERSION="2.0"> <LOGIN USER_LOGIN="adminname" PASSWORD="password"> <RIB_INFO MODE="write"> <FORCE_FORMAT VALUE="all"/> </RIB_INFO> </LOGIN> </RIBCL> The iLO 4 firmware version must be 2.44 or newer. Download the most recent version available on HPE.com as follows: The latest version of the iLO 4 firmware can be downloaded: iLO 4 firmware Click the "Revision History" tab to see the latest available version. NAND Format Methods This document describes five methods which can be used to format the NAND on a ProLiant Gen8 or Gen9 server: From the iLO 4 GUI (requires iLO 4 firmware version 2.61 or newer) From the Onboard Administrator (for servers in HPE BladeSystem c3000/c7000 Enclosures only) From Windows OS (using the HPQLOCFG.exe utility) From Windows PowerShell (using HPE iLO cmdlets for PowerShell) From Linux or VMware (using the hponcfg utility) NOTE: The steps to confirm that the NAND format was successful are shown at the end of this document. Additional considerations before performing a NAND format A NAND format can be performed while the server is online in most cases. Exception: For VMware ESXi hosts booting from the Embedded SD Card - it is strongly recommended to perform the NAND format with the ESXi OS shut down. This recommendation also applies when updating the iLO 4 firmware or resetting the iLO for ESXi hosts booting from the Embedded SD Card. A server AC power removal may be required (prior to the NAND format) in order for the NAND format to be successful. This can be accomplished for ML or DL-series servers by shutting down the server and disconnecting the power cables for a few seconds. For server blades, an E-fuse reset can be accomplished by logging into the OA CLI and typing "reset server#" where "#" is the bay number of the server blade. An AuxPwrCycle feature was added in iLO 4 firmware version 2.55 so that the equivalent of an AC power removal can be performed remotely on a server. Required steps to perform after a successful NAND format If the server is powered on when performing the NAND format, a server reboot is required after a successful NAND format. This reboot repopulates the RIS and RESTful data on the server during the next server POST with iLO 4 firmware version 2.53 or newer installed. If iLO 4 firmware version 2.50 or older is installed, connect to the iLO CLI (using putty) and use the following command "oemHP_clearRESTapistate" before rebooting the server to repopulate the RIS and RESTful data on the server. Server POST with iLO 4 firmware version 2.53 or newer installed: NOTE: If iLO 4 firmware version 2.50 or older is installed, connect to the iLO CLI (using putty) and use the following command "oemHP_clearRESTapistate" before rebooting the server to repopulate the RIS and RESTful data on the server. Reinstall Intelligent Provisioning (IP) to ensure that it is working properly and reported as installed in the iLO 4 GUI "System Information - Firmware Information" page. If using OneView: After the server is rebooted (to repopulate the RIS and RESTful data on the server), perform a server refresh in OneView; any existing errors in OneView will need to be marked as cleared after the refresh. NOTE: The screenshots below show the iLO firmware as version 2.60; however, these are for example purposes only. The actual firmware version is 2.61. NAND Format Method 1: Use the iLO 4 GUI iLO 4 firmware version 2.61 has added these new features for Gen8 / Gen9 servers: The iLO 4 GUI login screen displays a warning if there is an iLO Seft-Test error for the Embedded Flash/SD-CARD The iLO 4 GUI Overview and Diagnostics pages have added an "iLO Health" status The iLO 4 GUI Diagnostics page has a "Format Embedded Flash and reset iLO" button ( NOTE: the button is only visible if the "iLO Health" has a "Degraded" status) Steps to perform a NAND format via the iLO GUI 1. The iLO 4 GUI login will display a warning if the iLO has reported a self-test error on the Embedded Flash/SD-CARD. 2. The iLO 4 GUI Overview page will show the "iLO Health" has a "Degraded" status; click on "iLO Health" hyperlink to go directly to the iLO GUI Diagnostics page. 3. The iLO 4 GUI Diagnostics page will show a "Degraded" status for "Embedded Flash/SD-Card"; click on the "iLO Health" hyperlink and the "Format Embedded Flash and reset iLO" button becomes visible ( NOTE: the "iLO Health" hyperlink is only visible when the "ilO Health" has a "Degraded" status). 4. Click the "Format Embedded Flash and reset iLO" button to perform a NAND format. 5. A pop-up message is displayed as a final confirmation before performing a NAND format. 6. After the NAND format is initiated, the iLO 4 is automatically reset and the iLO 4 GUI login page is displayed (the user will need to log into the iLO 4 GUI again). NOTE: due to browser caching, the iLO 4 self-test warning is still displayed. NAND Format Method 2: Via Onboard Administrator (server blades only) 1. Log into the OA CLI as administrator using putty 2. Edit the entry below for the proper server bay number, then cut and paste the text into the putty session and observe the output to confirm the input was accepted by the OA. Copy the modified text in its entirety and paste in the OA CLI to initiate the NAND format. Syntax: HPONCFG <bay #> << end_marker <RIBCL VERSION="2.0"> <LOGIN USER_LOGIN="adminname" PASSWORD="password"> <RIB_INFO MODE="write"> <FORCE_FORMAT VALUE="all"/> </RIB_INFO> </LOGIN> </RIBCL> end_marker Here is an example (format bay 3 - bolded): HPONCFG 3 << end_marker <RIBCL VERSION="2.0"> <LOGIN USER_LOGIN="adminname" PASSWORD="password"> <RIB_INFO MODE="write"> <FORCE_FORMAT VALUE="all"/> </RIB_INFO> </LOGIN> </RIBCL> end_marker NAND Format Method 3: Windows (Remote Format) Download and install HPQLOCFG on the local server or a remote Windows host or client. To download the utility: https://support.hpe.com/connect/s/softwaredetails?language=ja&softwareId=MTX_2fed77de04c04076a0d453326d&tab=revisionHistory Place the Force_Format.XML file (described in the prerequisites section above) in the folder with the utility. The default location is C:\Program Files (x86)\Hewlett-Packard\HPLights-OutConfiguration Utility. Format the NAND using HPQLOCFG. The commands below assume the .xml file is named Force_Format.xml. The bold items in the commands need to be replaced with the iLO IP address, username, and password as shown in the screenshots. To format the local server NAND, use the following command: C:\Program Files (x86)\Hewlett-Packard\HPLights-OutConfiguration Utility>HPQLOCFG.exe-fForce_Format.xml-s <IPofiLO> To perform a remote server NAND format, use the following command: C:\Program Files (x86)\Hewlett-Packard\HPLights-OutConfiguration Utility>HPQLOCFG.exe -s <IPofTargetiLO> -u <iLO-Username> -p <Password-For-iLO> -f Force_Format.xml NAND Format Method 4: Windows PowerShell (Remote Format) Download and install "Scripting Tools for PowerShell: iLO cmdlets" (v1.5.0.1 or prior), which is available from these links: NOTE: "Scripting Tools for PowerShell: iLO cmdlets" versions v2.x and v1.x can both be installed on a server at the same time; version v2.x does not include the "Invoke-HPiLORIBCLCommand", so version v1.x must be installed and used in order to provide this capability. Scripting Tools for Windows PowerShell (x64): iLO cmdlets Version: 1.5.0.1(9 Mar 2018) File name: HPiLOCmdlets-x64.exe (1.1 MB) https://support.hpe.com/hpsc/swd/public/detail?sp4ts.oid=1008862655&swItemId=MTX_2eda1acc832d48ad944f694f12&swEnvOid=4184 Scripting Tools for Windows PowerShell (x86): iLO cmdlets Version: 1.5.0.1 (9 Mar 2018) File name: HPiLOCmdlets-x86.exe (1.1 MB) https://support.hpe.com/hpsc/swd/public/detail?swItemId=MTX_dd9188b9396745ce901012120b NOTE: The PowerShell commands below assume the Force_Format.xml file (described in the prerequisites section above) is in the C:\RIBCL_XML\ folder and named Force_Format.xml. Open a PowerShell Command line and enter the following commands (italicized). The iLO IP, username, and password need to be included in the second command. PS C:\>$xml = ([string](get-content "C:\RIBCL_XML\Force_Format.xml")) PS C:\>Invoke-HPiLORIBCLCommand -server iLOIP -Username Administrator - P assword password -RIBCLCommand $xml -DisableCertificateAuthentication A status similar to the following will be displayed if the commands are accepted: IP HOSTNAME STATUS_TYPE STATUS_MESSAGE 10.1.1.30 ilo123456.iLO.local WARNING Forcing a format of the partition after the iLO reset. NAND Format Method 5: Linux or VMware (Local Format) Download and install HPONCFG on the local server or a remote Windows host or client. To download the utility: Click the following link: https://support.hpe.com . Enter a product name (e.g., hponcfg) in the text field under Enter a Product Name or Number. Click Go. Select the appropriate product model from the Results list (if prompted). Click the "drivers, software & firmware" hyperlink under the Download Options tab. Select the system's specific operating system from the Operating Systems dropdown menu. Click the category Software under the software type dropdown menu. Select the latest release of the HP Lights-Out Online Configuration Utility for Linux (or HPE ESXi Utilities Offline Bundle for VMware). Click Download. Place the "Force_Format.xml" file (described in the prerequisites section above) in the folder with the utility. The default location is /opt/hp/tools. Format the NAND usinghponcfg from the /opt/hp/tools directory. The command below assumes the .xml file is named Force_Format.xml. "./hponcfg -f Force_Format.xml" When the format completes successfully, there will be a message similar to the following: "Script succeeded for IP:XXX.XXX.XXX.XXX." How to determine if the NAND format was successful (when iLO 4 firmware version 2.61 or newer is installed): After the NAND format is initiated, the iLO 4 is automatically reset and the iLO 4 GUI login page is displayed (the user will need to log into the iLO 4 GUI again). Perform these checks to confirm whether the NAND format was successful: Refresh the browser cache to confirm that the iLO 4 GUI login page no longer displays a self-test warning: Confirm the iLO Health is OK on the iLO 4 GUI Overview page: Look for the following message will be in the iLO Event Log to confirm a successful NAND format: "Embedded Flash/SD-CARD: One or more storage devices have been formatted" Open iLO 4 Diagnostics page to confirm the iLO Self-Test status is normal (not degraded). Confirm the iLO 4 GUI Diagnostics page will show the "iLO Health" has a "green" status ("iLO Health" is no longer a hyperlink) with the 2.61 firmware installed: NOTE: If the NAND format was not successful, additional steps will be required to resolve the iLO 4 self-test error for the Embedded Flash/SD-CARD. Refer to this advisory for further details: Customer Advisory c04996097, (Revision) - HPE Integrated Lights-Out 4 (iLO 4) - HPE Active Health System (AHS) Logs and HPE OneView Profiles May Be Unavailable Causing iLO Self-Test Error 8192, Embedded Media Manager and Other Errors:" Find the document at: https://support.hpe.com/hpesc/public/docDisplay?docId=emr_na-c04996097
