Operational Defect Database
BugZero updated this defect 58 days ago.
Data sources
All data on this page is proprietary to BugZero® or gathered from public sources
VMware | 445046
Smarts: How to generate a core file in Linux
Last update date:
3/22/2024
Affected products:
Smart Assurance - SMARTS
Affected releases:
No affected releases provided.
Fixed releases:
No fixed releases provided.
Description:
Symptoms
The Smarts server is crashing, and I need to create a Memory dump on Linux server for troubleshooting by EMC Customer Service.This article describes two methods to take a memory dump in LINUX. One requires the application to crash before the memory dump will happen, and one allows you to force a memory dump while it remains running. NOTE: In some cases, you may not be able to generate a core file in Smarts IP/Smarts SAM environment due to the fact that gcore is not installed on Linux server. If this symptom exists, you may see this log file entry: Cannot run gcore -o sm_gcore.out <pid> command in Smarts IP/Smarts SAM environment because gcore not installed on the Linux server.
Resolution
To facilicate a LINUX core dump, follow these steps: Stop the Smarts server: (NOTE: These example commands are only applicable to a SAM server.) Running from the command line:<BASEDIR>/smarts/bin/sm_server -n InCharge-SAM -c ics -b localhost:427 --ignore-restore-errors --output --daemonTo confirm it's stopped, execute:<BASEDIR>/smarts/brcontrolWhen it's installed as a service:<BASEDIR>/smarts/bin/sm_service stop InCharge-SA1. Enable core file dumping To confirm the service is stoppped, execute:<BASEDIR>/smarts/sm_service show We now set the memory usage to unlimited to make sure our Smarts server is not simply being killed by LINUX for consuming more memory than the set limit. In the same putty window, run this LINUX command: <BASEDIR>/smarts/ulimit -c unlimited Reload the SAM server in the same Putty sesion, or command line window. Running from the command line:<BASEDIR>/smarts/bin/sm_server -n InCharge-SAM -c icf --bootstrap=bootstrap-am-pm.conf -b localhost:427 --output --daemonTo confirm the domain has started, execute this command:<BASEDIR>/smarts/bin/brcontrolWhen it's installed as a service:<BASEDIR>/smarts/bin/sm_service start InCharge-SAMTo confirm the domain has started, excecute this command:<BASEDIR>/smarts/bin/sm_service show Take the steps necesary to reproduce the hang. Or, if need be, wait for the issue to reproduce.Collect core file from bin directory. We also suggest collect the rps file. Upload the files to the EMC support case. The rps file will be called by the same name as the domain name, and be stored in the <BASEDIR>/smarts/local/repos/icf folder. The memory dump fies should be left in the same folder that you started the Smarts domain in. However, if you started it as a service here are some proposed folder locations you may find them in: <base dir>/SAM/smarts/bin<base dir>/SAM/smarts/local/logs/home/usr To cause a memory dump to happen at will, follow one of these two suggestions. Running GDB: (gdb) attach <pid>(gdb) generate-core-file <optional-filename>(gdb) detachNOTE: If you cannot generate a core file because gdb is not installed, you can use "gdb" which is shipped with Smarts products (located at <Base dir>/smarts/bin/gdb) to generate a core file. This can be done as follows: Edit the following commands into a file called gcore.gdb:set pagination offset width 65536gcoredetachquit Run the following command:smarts/bin/gdb -x gcore.gdb -p <pid> Using the kill signal command: kill -s SIGABRT <pid> NOTE: Look for the memory dump files in the folder where the application was started from, or from these locations:<base dir>/SAM/smarts/bin<base dir>/SAM/smarts/local/logs/home/usr
Related Information
What is the .gdb file in <basedir>smarts/bin is used for and can I remove it? Smarts needs gdb to dump stacktrace in real time . When smarts hits an exception (either fatal or error) or receives a signal, the exception or signal handler would invoke the included gdb to create stacktrace into log file.If gdb is removed, stacktrace information will not be logged in the log file if the system ever crashes. The included gdb has been tested to work with smarts to generate stacktrace. And if the native gdb is not on the search path of smarts runtime, smarts may not be able to find gdb if the included one is removed. How to test for the current ulimit and core dump setting:1: Search your logs: U can search your current SAM, or IP log file for this text - "Current limit on CORE file"In a linux environment , you would use the "VI" editor and search for the above text, as per the below example. vi <basedir>smarts/local/logs/<Name of SAM log file_en_US_UTF-8.log> Here an example of what you'd see: Current limit on CORE file size: 15.999 EiB (18446744073709551613)Current address space limit: 15.999 EiB (18446744073709551613)Current file descriptor limit: 256Current top of heap: 43002183682: Using the linux command line environment: Check the OSs setting for name of the core files by [root@itops-tco-229 user1]# cat /proc/sys/kernel/core_pattern If you only see the name "core" here, it indicates that this will be the name of the core file name and the default location will be from where process is executed. Check the ulimit settings [root@itops-tco-229 user1]# ulimit -c unlimited [root@itops-tco-229 juser1]# ulimit -a An example result from thes commands is below: core file size (blocks, -c) unlimited data seg size (kbytes, -d) unlimited scheduling priority (-e) 0 file size (blocks, -f) unlimited pending signals (-i) 139264 max locked memory (kbytes, -l) 32 max memory size (kbytes, -m) unlimited open files (-n) 1024 pipe size (512 bytes, -p) 8 POSIX message queues (bytes, -q) 819200 real-time priority (-r) 0 stack size (kbytes, -s) 10240 cpu time (seconds, -t) unlimited max user processes (-u) 139264 virtual memory (kbytes, -v) unlimited file locks (-x) unlimited
