Investigating and troubleshooting dashboard metric, rule, and alarm issues (multiple causes) (4308244)

• database agent rules and alarms not firing
• metrics are showing the incorrect values
• data is missing from the dashboards or alarms
• alarms are delayed
• duplicate alarms are fired.

Cause 1

Cause 18

Alarm is disabled in the alarm template that is assigned to the target.

In situations where data is shown incorrectly in a dashboard or an alarm misfires, the first step in investigating is to differentiate whether the timeouts/missing data is caused because by a collection issue, a configuration setting, the rule, or the FMS itself.

The general flow would be:

1. Note the Agent UI for missing data or timeouts per specific screen/collection
2. Review the metrics collected in the Foglight panels or in the topology in the context of understanding the collection's purpose.
3. Check the Agent log and to look for a failing collection
4. Check the agent configuration
5. Review the alarms and historical alarms
6. Review the rule used to fire the alarm

Metrics

All metrics are collected from the Monitored Host and and submitted to the FMS, registered to Metric objects.

A Metric is an object in the FMS that holds the data in several data segments:

• History segment - which contains the whole information collected for that metric
• Latest segment - contains the latest entry for the metric, regardless of the selected time period
• Period segment - contains the Average data (value) of the metric. For example CPU usage calculated average according to the selected tome period
• Current segment - which has the data of the last entry in the metric, per the selected time period
  The segments of Latest, Period & Current are a breakdown of the History general segment of the metric.

The Current data segment and the Latest data segment entries are the same when the user chooses to see the "Last...", like "Last hour" for the selected time period.

INVESTIGATION: To investigate "raw" metrics, users can drill down into the FMS topology through various means including Configuration | Data or Administration | Tooling | Script Console
 

Collections

A collection is typically a SQL statement or OS command(s) that are running by a database agent from the FglAM against the monitored host.

• If the data is successfully retrieved by the agent, typically no messages will be printed to the database agent log unless the FglAM is running in "debug" mode.
• If there is a collection failure or an error is encountered, then a log error message is printed to the Agent log file indicating a proximal cause such as a query timeout, data issue (such as variable size), and incorrect permissions.

Agent Managers (and some agent types) can be run in debug mode to write additional details to the agent log files.

INVESTIGATION: Review the database agent log file to look for an query errors that correspond to the collection which is being investigated (e.g. Top SQL). Analogous SQL or OS queries can also be run directly against the monitored host to see the results outside of Foglight.

Agent Configuration

Each agent is managed by the settings configured in the Agent Status Properties for each agent. These settings include

• connection details (e.g. hostname, user id, password, SSL)
• collection frequencies (timeouts and how often the collection runs)
• other collection specific values (e.g. batch size) or flags (enable/disable a setting)

Many database agents (e.g. Oracle, SQL Server, DB2) list all of the agent configuration settings each time a database agent is restarted.

INVESTIGATION: Review the current settings for the agent in question and make relevant adjustments such as

• increasing the amount of rows collected
• increase or decrease the frequency of how often the collection runs
• increasing the length of time before a query times out
• use the validate connection option in some agents to reset the username and password 

Alarms

If a rule processes the Metric data and determines that the thresholds point to exception (for example, CPU usage is over 80, and 80 is the threshold which beyond an Alarm should be fired), an Alarm will be triggered.

According to each Rule definition (thresholds, Baseline thresholds, Boolean), an Alarm will be fired.

Rule with thresholds/Baseline thresholds can trigger an Alarm with the below severities:

Those are Multiple Severity Alarms.

• Rule which is based on Baseline thresholds will fire according the Baseline thresholds.
  Baseline thresholds have their own learning curve and inner algorithm and cannot be modified.

Baseline Alarms are considered as Multiple Severity Alarms.

• Besides the Multiple Severity alarms, there are the Simple Rule alarms.
  those alarms have only one severity defined, and the color will be one as well (yellow, in most alarms).

INVESTIGATION: Review the details of the alarm messages itself and look at the history of the alarms to identify any patterns. 

Rules

The data gathered in each of the Metrics and its data segments is constantly being checked by the Rules.
A rule is defined per each metric, and the rule checks the metric data in order to alert the user, if required.
The alert is in the form of Alarm and mail notification (if configured).
In order for the Rule to "know" if an Alarm should be triggered, the Rule uses either thresholds or Boolean condition.

The types of Rules are:

• Rules that are checking strict thresholds numbers
• Rules that are checking Baseline thresholds
• Rules that are checking Boolean conditions (like yes/no, enabled/disabled)

INVESTIGATION: Review the design of the rule and compare the rule to an "out of the box" copy of the rule to identify a rule has been modified by a user. If they rule has been customized, restore the rule back to its original design to confirm that any issues still exist with the rule. As well, review and adjust the thresholds used by rules to trigger alarm conditions.
 

Other relevant issue resolutions 

Resolution 2

Enable the Alarms Email Notifications as per KB 4310657.

• Review both the FMS and FglAM Memory allocation and Java Heap assignments (-Xmx/-Xms), and adjust per the applicable Foglight product guides.
• Restart the FMS process, or 
• Reboot the FMS, or
• Failover the FMS if HA is configured, then fail back to the expected Primary node if a service outage is not possible at the time.

Resolution 7

Confirm if there is an active Blackout set for the instance or Agent

• If yes, delete the blackout from the Global View Databases dashboard via Settings | Manage Alarm Blackouts or from Administration | Setup | Blackouts | Manage Agent Blackout  then Remove the blackout for that instance.

Resolution 8

Please refer to KB 4295903 on counting and purging alarms

Resolution 9

Resolution 18

Enable the alarm in the alarm template; refer to Editing Alarm Templates (4226517) for more information on how to disable or enable the alarms.