-
Título
High CPU usage on Foglight Agent Manager (FglAM) running database agents (where no PI repository is installed on the same FglAM) -
Descrição
There is high CPU utilization up to 100% on the Foglight Agent Manager (FglAM) where the database agents are running.
There is no PI repository installed directly on this server.
-
Causa
* Sizing issues are the most commons causes for performance issues on Foglight Agent Managers *
Cause 1
Insufficient sizing (CPU, memory, disk)
Cause 2VMWare tools were not running and up to date. During investigation the FglAM was also using only 2.5 GB memory but had much more assigned.
Cause 3Issue with a specific collection or connection for one or more agents running on the FglAM. One or more agents could cause the CPU for the FglAM to spike.
For example, among Oracle agents on hosts with a high number of disks (~16K) on hosts 75-1 / 7-2, the Filesystem IO collection takes extensive resources to run iostat on the monitoring databases.
Cause 4Third-party DataDirect JDBC driver conflict with the SSL connection using Java 8.
Cause 5Simultaneous running of collections on numerous agents
Cause 6Performance impact of MongoDB, MySQL, SSAS, or PostgreSQL database agents
Cause 7Antivirus running on the FgLAM
Cause 8Undetermined Microsoft driver
Cause 9Defect FOG-3307
Cause 10FglAM issue with 6.1.0 release.
-
Resolução
Resolution 1
* This is the most common resolution to this issue *
Review the Foglight for Databases deployment / sizing guide to confirm that the current FglAM sizing requirements including the number of CPUs are met
Resolution 2Confirm that the VMWare tools are running on the Virtual Machine where the FglAM is installed and that the VMWare tools are up to date.
Resolution 3Turn off agents in groups and/or individually to identify with agent may be causing the performance issue. Once an agent is identified either recreate or edit the agent status properties to fix the underlying issue.
For example, in Oracle agents use the steps in KB article 4308784 to set the Filesystem IO collection frequency to 600 to 900 seconds (10 to 15 minutes) or disable the Filesystem IO collection.
Resolution 4Remove any override references to the older Data Direct driver from the baseline.jvmargs.config file (-Dsqlserver.driver.class.name=com.dell.dsgi.jdbc.sqlserver.SQLServerDriver) and refer to KB 4295822 regarding changing the SSL and TLS settings for SQL Server agents.
Resolution 5When restarting a FglAM that has hundreds of agents activated, the agents will typically all activate at the same time and then run their collections together. This can place a lot of load on a FglAM server. Many database agent collections have a 300 second interval (offline) which are roughly run in the same order. This means that intensive collections like DBSS_Databases might run on all of the agents around the same time every five minutes.One suggestion is to start up the DB agents in batches of 20-30 agents, about every 17 seconds apart… or we could add the following JVM flag in the baseline.jvm.config file, then restart the FglAM.vmparameter.0 = "-Dagent.collector.schedule.load.max.delay.millis=300000";This will spread agent activation over 5 min window (default is 2min, default collection cycle is 5min)
Resolution 6Deactivate any of the following types of database agents running on the agent manager.
- SSAS (SQL Server Analysis Services)
- MySQL
- PostgreSQL
- MongoDB
If deactivating the agents has been found to improve the FglAM stability then these agents should be deleted and recreated on a separate FglAM.
Resolution 7Disable any antivirus running on the FglAM for about five minutes to determine if there is a performance impact
Resolution 81). Stop FglAM
2). Edit the baseline.jvmargs.config file in the {fglam}\state\default\config folder to add a JVM (Java Virtual Machine) flag to change the driver used by the FglAM from the newer Microsoft Driver back to the Data Direct driver that used for several earlier Foglight versions. The value X should be incremented to one number higher than the highest vmparameter line in the file.
vmparameter.X = "-Dsqlserver.driver.class.name=com.dell.dsgi.jdbc.sqlserver.SQLServerDriver";3). Restart FglAM
4). Wait 5-10 minutes for the metrics to appear and the CPU to stabilize.
Resolution 9
Upgrade the SQL Server cartridge to 6.1.0.10 or higher.
Resolution 10Please contact Support for a hotfix for this issue in FglAM version 6.1.0.
Troubleshooting suggestions- Choose one FglAM and disable SQL PI from the agents on that FglAM, this would narrow the issue down to being SQL PI collections causing the problem (i.e. if the PI extension is not connected to any agents on that FglAM, then everything is caused by a database agent collection for one or more agents)
- Manually shut down ½ of the agents on the FglAM to see if the CPU drops. If it does not, then shut down half of the remaining agents, and try again, etc. Alternatively, start up the agents in groups of 20-30, then wait five minutes to see if any group causes the spike and the CPU doesn’t return to normal.