-
Title
Investigating and troubleshooting rule and alarm issues (multiple causes) -
Description
Generic steps to troubleshoot and investigate database agent rule and alarm issues.
Database agent rules and alarms not firing.
Alarms are fired with the incorrect values.
Alarms are delayed.
Duplicate alarms are fired.
-
Cause
Cause 1
Disable alarm in the database agent administration panel
Cause 2
Threshold is too high for a rule to fire.
Cause 3
Email is not configured or is disabled in the agent administration panel
Cause 4
Alarm sensitivity is set for some alarm severities not to be displayed.
Cause 5
Disabled rule(s) in the Rules Management dashboard
Cause 6
Alarms Service is stopped.
Cause 7
FMS lacks sufficient resources to process monitoring data or generate Alarms or Email Notifications.
Cause 8
No matching data has been collected to trigger an alarm.
Cause 9
Blackout is configured
Cause 10
High number of alarms in the alarms table
Cause 11
Email received is from a different FMS
Cause 12
Customized or multiple (custom) copies of the same rule.
Cause 13
Unsuccessful connection to host or database
Cause 14
Collection frequency is set higher than normal or the event occurs between agent collections
Cause 15
Email server settings not configured
Cause 16
Oracle Alert log filtering or SQL Server Error log filtering for individual and summary alarms is set to only fire Fatal or OFF.
Cause 17
Defect ORAFOG-681
Cause 18
Rule is cloned from another rule and is misconfigured or cannot be managed using the Database Administration UI.
Cause 19
Data issues from the query used by collection
Cause 20
Some issues may have been temporary, such as due to a long running query, intensive use of TEMP tablespace, or a tablespace autoextended after the alarm fired.
-
Resolution
Resolution 1
Enable the alarm as per KB 103065
- Navigate to Databases | Databases
- Select the database for which the alarm was not triggered
- Click Settings | Administration
- On the Administration dashboard, click on the Alarms
- Search for the alarm and check if this is active and enabled.
- Click on "Set" and "Save changes".
Resolution 2
Refer to KB 93998 for details on changing thresholds for database monitoring agents.
* For testing purposes, temporarily set the threshold to a very low value (for example 1%) to confirm that the alarm is working and fires.
Resolution 3
Enable the Alarms Email Notifications as per KB 90935 and KB 261762 (video).
Resolution 4
Review KB 232091 to configure the alarm sensitivity to fire the appropriate severity levels.
* For testing purposes, set the agent to tuning, and confirm that alarms are fired.
Resolution 5
Enable the rules in the Rules Management dashboard by navigating to Administration | Rules & Notifications | Rules
Resolution 6
Restart the FMS process
or
- (Re)Start the Alarm Service via the JMX console while logged in as 'Foglight' user:
- Browse to: http:///jmx-console/HtmlAdaptor?action=inspectMBean&name=com.quest.nitro%3Aservice%3DAlarm
- Search for and Invoke the void stop(), then void start() Mbeans.
Resolution 7
Options include:
- Review both the FMS and FglAM Memory allocation and Java Heap assignments (-Xmx/-Xms), and adjust per the applicable Foglight product guides.
- Restart the FMS process, or
- Reboot the FMS, or
- Failover the FMS if HA is configured, then fail back to the expected Primary node if a service outage is not possible at the time.
Resolution 8
It's possible on very under-utilized systems that no alarms are fired due to the configured Rule/Alarm criteria or simply that Alarm thresholds are not met. Also If the agents are stopped or are unable to collect data alarms will never fire, except for Agent Health, Credential, Availability or Connection specific rules. Review the Agent dashboards for current expected data.
Resolution 9
Confirm if there is an active Blackout set for the instance or Agent as per KB 129995 and 155621
- If yes, the blackout from Database | Settings | Manage Alarms Blackouts or Manage Agent Blackout | Remove the blackout for that instance.
As a test, drill down on the instance from the Database dashboard | click on Activity | Real Time | check the Availability to see if there is blackout alert.
Resolution 10
Please refer to KB 80646 for more information on counting alarms in the alarms table.
Please refer to KB 66577 for more information on purging the alarms table.
Resolution 11
If the customer receives alarms and the alarm or rule has been disabled, please confirm if there is a second Foglight Management Server (FMS) environment.
Resolution 12
Rule has been customized (emailaction, fire multiple times, copied rule). Reviewing the rule name and the alarm format and email text to look for non-default text can be an indication that a rule has been customized.
Compare the rule conditions to the same rule on a FMS system that has not been modified and uses the same cartridge version.
Resolution 13
Validate connectivity to the host as per KB 114121 (Oracle), 234887 (SQL Server) and 234154 (DB2, see Resolution #1)
Resolution 14
Set the database agent collection for the alarm metric to query the host more often. Refer to KB 177914 for details on changing database agent collection frequencies.
Resolution 15
Check the SMTP and email configuration on the FMS server as per KB 69979.
Resolution 16
Set the SQL Server Error Log Filter or Oracle Alert Log Filtering to fire for Warning alarms.
Resolution 17
A fix specific for the "DBO Usability Connect Availability" rule / alarm is available in the Oracle 5.9.2.1 and higher cartridge versions. Please consult KB 259302 for more details.
Resolution 18
Enable and configure the original "out of the box" rule using the database agent administration panel.
Resolution 19
Run a query similar to the database agent's collection and compare the results to the alarm message.
Resolution 20
Access the database agent's raw topology metrics using the Configuration | Data pages and choose a time range in the zonar that corresponds to when the alarm was fired and review the actual data to identify any temporary changes in the host system (e.g. afterhour downtime, network issues, high IO due to backups).