How to generate a JVM heap dump and histogram (63622)

Sometimes it is necessary to collect heap dump on the FMS to analyze JVM functionality.

For Versions 5.6.11 - 5.7.5, Run the following command:

There are two methods to generate a heap dump.

• Manually
• Automatically

Manually:

1. Install Java JDK that matchs the JRE version of the FMS for your OS / bit version.
Foglight ships with JRE, not JDK.
Look in the Foglight Management Server log for this line: org.jboss.system.server.ServerInfo
This will tell you what version to download (be sure to look for JVM and not OS details)

Download JDK for the platform:

• Select the appropriate Java version
• Select your appropriate O/S version
• Select the Java SE Development Kit

Install on your FMS in a separate folder outside of $FGLHOME.

2. Wait for the Foglight memory to start to climb (ie. perf report shows no garbage collection)

3. Idenitfy the FMS executable process ID (PID) and generate a JDK Histogram
<JDK_HOME>/bin/jmap -histo -F <fms pid> > histrogram.txt

IF DIRECTED BY SUPPORT, continue to gather the full JDK Heap dump

4. Generate a JDK Dump using the Foglight PID.
 <JDK_HOME>/bin/jmap -dump:live,format=b,file=app.bin <PID>

This will create a dump file named app.bin in the current folder where the command was executed.

5. Generate a fresh FMS support bundle (if possible)

6. Submit all items of information to Quest Support. If any of the files are > 10mb request a FTP link to upload.

Starting with Foglight Management Server (FMS) version 5.7.5.3 and above, jvm heap histogram can be generated via command line:

on Windows platform:

fms --heap-histogram

on *NIX platform:

./fms --heap-histogram

Automatically:

1. Edit the $FGLHOME/config/server.config file, search for the appropriate server.vm.option number and add the entry:
     server.vm.option2 = "-XX:+HeapDumpOnOutOfMemoryError";

2. Save the file

3. Restart the FMS software

4. When an OOM condition occurs, a java heap dump will be generated.

FglAM ability to generate a thread dump via cmd line switch

While the FglAM is running, one can now pass a cmd line argument to generate a heap dump while a problem is being observered.

1) $FglAM_HOME/bin/fglam --heap-dump {filename}

ie. $FglAM_HOME/bin/fglam --heap-dump heap.dump

2) This will create the file in the $FglAM_HOME/client/{version} folder.
     ie.  C:\Quest_Software\Foglight_Agent_Manager\client\5.7.0-201405021222-334052-588\heap.dump

Alternatively, in situations where trying to troubleshoot FglAM crash due to heap OOM  error condition, add below lines in baseline.jvmargs.config file and then restart Agent Manager.

vmparameter.0 = "-XX:+HeapDumpOnOutOfMemoryError";

vmparameter.1 = "-XX:HeapDumpPath=<path_to_foled_where_file_should_be_created>";

(if 0 ,1 are alread used for other parameters then use the next available number)

NOTE:  If the dump file exists from a previous run, the cmd will not overwrite an existing file.  The previous file dump must be deleted.

Orignal instructions

1. Install Java JDK that matchs the JVM version of the FMS

2. Run the jmap utility by typing jmap -dump:file=app.bin <PID> where PID is the process id of the FMS process

3. This will create a dump file named app.bin in the current folder.