High CPU usage on Foglight Agent Manager (FglAM) running database agents (where no PI repository is installed on the same FglAM) (4232005)

There is high CPU utilization up to 100% on the Foglight Agent Manager (FglAM) where the database agents are running.

There is no PI repository installed directly on this server.

* Sizing issues are the most commons causes for performance issues on Foglight Agent Managers *

Cause 1 

Insufficient sizing (CPU, memory, disk)

Cause 2

VMWare tools were not running and up to date. During investigation the FglAM was also using only 2.5 GB memory but had much more assigned.

Cause 3

Cause 5

Simultaneous running of collections on numerous agents

Cause 6

Performance impact of MongoDB, MySQL, SSAS, or PostgreSQL database agents

Cause 7

Antivirus running on the FgLAM

Cause 8

Undetermined Microsoft driver 

Cause 9

Resolution 1

* This is the most common resolution to this issue *

Review the Foglight for Databases deployment / sizing guide to confirm that the current FglAM sizing requirements including the number of CPUs are met

Resolution 2

Confirm that the VMWare tools are running on the Virtual Machine where the FglAM is installed and that the VMWare tools are up to date.

Resolution 3

Turn off agents in groups and/or individually to identify with agent may be causing the performance issue. Once an agent is identified either recreate or edit the agent status properties to fix the underlying issue.

For example, in Oracle agents use the steps in KB article 4308784 to set the Filesystem IO collection frequency to 600 to 900 seconds (10 to 15 minutes) or disable the Filesystem IO collection.

Resolution 4

Remove any override references to the older Data Direct driver from the baseline.jvmargs.config file (-Dsqlserver.driver.class.name=com.dell.dsgi.jdbc.sqlserver.SQLServerDriver) and refer to KB 4295822 regarding changing the SSL and TLS settings for SQL Server agents.

Resolution 5

vmparameter.0 = "-Dagent.collector.schedule.load.max.delay.millis=300000";

This will spread agent activation over 5 min window (default is 2min, default collection cycle is 5min)

Resolution 6

Deactivate any of the following types of database agents running on the agent manager.

• SSAS (SQL Server Analysis Services)
• MySQL
• PostgreSQL
• MongoDB

If deactivating the agents has been found to improve the FglAM stability then these agents should be deleted and recreated on a separate FglAM.

Resolution 7

Disable any antivirus running on the FglAM for about five minutes to determine if there is a performance impact

Resolution 8

2). Edit the baseline.jvmargs.config file in the {fglam}\state\default\config folder to add a JVM (Java Virtual Machine) flag to change the driver used by the FglAM from the newer Microsoft Driver back to the Data Direct driver that used for several earlier Foglight versions. The value X should be incremented to one number higher than the highest vmparameter line in the file.

Resolution 9

Upgrade the SQL Server cartridge to 6.1.0.10 or higher.

Resolution 10

Please contact Support for a hotfix for this issue in FglAM version 6.1.0.

Troubleshooting suggestions

• Choose one FglAM and disable SQL PI from the agents on that FglAM, this would narrow the issue down to being SQL PI collections causing the problem (i.e. if the PI extension is not connected to any agents on that FglAM, then everything is caused by a database agent collection for one or more agents)
• Manually shut down ½ of the agents on the FglAM to see if the CPU drops. If it does not, then shut down half of the remaining agents, and try again, etc. Alternatively, start up the agents in groups of 20-30, then wait five minutes to see if any group causes the spike and the CPU doesn’t return to normal.