Chat now with support
Chat with Support

Foglight 5.9.1 - Performance Tuning Field Guide

Overview Hardware and Operating System Tuning Management Server Tuning Java Virtual Machine Tuning Backend Database Tuning High Availability (HA) Tuning Agent Tuning Appendix: Performance Health Check Appendix: Analyzing a Support Bundle

Management Server Garbage Collectors

There are two types of garbage collectors:

1
ParNew—short and easy collections done in parallel, these have a low impact on the Foglight™ Management Server in normal operation.
2
ConcurrentMarkSweep—long and time-consuming collections that cause the JVM to pause everything else. The Management Server can appear to “freeze” during these cycles.

Examine the graphs for indications of issues.

Figure 5. Problems with the ParNew GC

The graph above indicates problems with the ParNew GC. The count (blue line) is up to 4, and the time (orange and brown) has increased to minutes.

Figure 6. Problems with the ConcurrentMarkSweep GC

The graph above indicates problems with the ConcurrentMarkSweep GC. The time (brown) has increased to minutes and the GC is called frequently.

These two graphs indicate the following issues:

•
GCs are frequent and consuming a large portion of CPU time.
•
High CPU usage is likely occurring.
•
Memory problems are present, which can be resolved by increasing the heap size and/or debugging where the heap is being used. Common sources for excessive heap usage are: derivation rulettes, cached metrics, and topology objects.
 
* 
NOTE: It is rare that the heap needs to be larger than 8 GB.

Increasing the heap size will likely seem to resolve an issue, but the root cause of the memory issue will soon grow to the new heap level, causing the issue to reappear.

Proper analysis of heap usage is necessary to ensure the root cause is resolved.

JDBC Connection Pool

The connection pool by itself does not indicate a problem, but can point to possible causes, such as: slow database, inefficient queries, or data intensive dashboards.

Figure 7. JDBC Connection Pool

If the available connections (orange line) flatlines at the bottom, the database connections are used for long periods of time, so no database connections can be made. The following error indicates no available connections:

blocking timeout ( 30000 [ms] ); - nested throwable:(javax.resource.ResourceException: No ManagedConnections available within configured blocking timeout ( 30000 [ms] ))

Keep this in mind while investigating other issues.

Derivation Rulette

A derivation rulette is an instance of a derived metric definition that is tied to a particular Topology Object. Derivation rulettes take memory to store. Depending on the Foglight™ Management Server version, anywhere from 4k (Management Server versions earlier than 5.5.5) to 1.5k (versions 5.5.5 and later). The more derivation rulettes you have, the more JVM heap is locked and cannot be freed, which leads to JVM heap exhaustion and performance problems.

Figure 8. Derivation rulette

Examine the count to determine whether there is an issue.

•
Counts < 100k should not have much of an impact.
•
Counts > 100k but < 1 million should be examined
•
Counts in the millions can use gigabytes of the heap all by themselves.

Examine the diagnostic snapshot to determine the source of the problematic rulettes.

Related Documents

The document was helpful.

Select Rating

I easily found the information I needed.

Select Rating