Foglight 6.3.0 - Administration and Configuration Guide

Foglight Agent Manager Performance Overview

The Overview view shows the numbers of remote hosts and Foglight agent instances that are connected to the Foglight Management Server. It also shows the maximal length of time spent by the Agent Manager (FglAM) on data polling.

The Message Processing tab shows tables for the number of pending messages and the number of incoming queue threads for a selected FglAM instance.

The Clock Skew tab contains a graph that displays the upstream difference per second for a selected FglAM instance.

The Bandwidth tab displays metrics for bandwidth usage samples taken at prescribed intervals per second for a selected FglAM instance, including the bytes in, bytes out, and real request time for a selected servlet. The bandwidth statistics reflect the bandwidth (bytes per second) used by the FglAM instance data over a specific time interval.

The Misbehaving Clients tab displays the disconnections per count for a selected FglAM instance.

The XML Serialization tab shows tables of the serialization rate and time for the message serialization to and from XML for a selected FglAM instance.

The CPU tab displays the amount of time the CPU associated with a selected FglAM instance spends executing active processes and their number. For example, a sudden increase in CPU time may indicate that the user code is running inefficiently or a possible runaway process. Also, high CPU loads sometimes suggest that the host needs more CPU power to run efficiently.

The Memory Usage tab displays the amount of available memory and swap space for a selected FglAM instance. For example, a shortage of swap space often suggests a memory shortage, while a decline in the available memory may indicate a memory leaking process.

The CDT Submission tab displays information about the complexity of data that agents collect and submit for Canonical Data Transform (CDT) processing. Excessively large values or sudden increases in this data can result in performance problems.

The following metrics are displayed for each agent:

Backing Up and Restoring Foglight

The following table shows the possible ways of backing up Foglight, some of which include the backup of the file system, and in some cases the system registry (on Windows). Each option shows a sequence of actions that can be performed to back up Foglight.

Table 10. Options for backing up Foglight.

Foglight Single Tier
	Foglight Management Server on Windows with embedded MySQL
Option 1			• Stop the Foglight Management Server. • Perform a full system backup, including the file system and the Windows Registry.
Option 2			• Issue a mysqldump command to export the MySQL database. Alternatively, use the InnoDB HotBackup tool to back up the embedded MySQL database. TIP: To restore the database after backing it up with the mysql command, use MySQL. To restore the database after backing it up with InnoDB HotBackup and applying the logs, shut down the Foglight Management Server, copy the backup files to <Foglight_home>/mysql/data, and start the Foglight Management Server • Perform a full system backup, including the file system and the Windows Registry, but excluding the MySQL data and transaction logs.
Option 3			• Issue a mysqldump command to export the MySQL database to a remote drive or a backup tape. TIP: This option does not require backing up the Foglight installation directory. To restore Foglight after choosing this backup method, you need to complete the following actions: 1. Install the Foglight Management Server and any cartridges that existed in the previous installation. 2. Shut down Foglight. 3. Delete the MySQL data files. 4. Start MySQL and import the exported dump file. 5. Shut down MySQL. 6. Start Foglight. Alternatively, instead of steps 3 through 6, use the InnoDB HotBackup tool to back up the embedded MySQL database.
	Foglight Management Server on Windows with embedded PostgreSQL
Option 1			• Stop the Foglight Management Server. • Use the zip tool to compress the installation directory of the Foglight Management Server. TIP: If you want to restore the database to a non-embedded PostgreSQL server, see the “Foglight Two Tier” section followed in this table.
	Foglight Management Server on Unix with embedded MySQL
Option 1			• Stop the Foglight Management Server. • Create a TAR or a ZIP file containing the Foglight installation directory and copy it to a remote drive or a backup tape. TIP: The ZIP format is recommended. TAR files can result in corrupted binary and MySQL data files.
Option 2			• Issue a mysqldump command to export the MySQL database to a remote drive or a backup tape. Alternatively, use the InnoDB HotBackup tool to back up the embedded MySQL database. • Create a TAR or a ZIP file containing the Foglight installation directory, excluding the MySQL data files, and copy it to a remote drive or a backup tape. TIP: The ZIP format is recommended. TAR files can result in corrupted binary and MySQL data files.
	Foglight Management Server on Unix with embedded PostgreSQL
Option 1			• Stop the Foglight Management Server. • Create a TAR or a ZIP file containing the Foglight installation directory and copy it to a remote drive or a backup tape. TIP: The ZIP format is recommended. TAR files can result in corrupted binary and PostgreSQL data files. The following is the example of backup commands: ZIP: zip -r Foglight.zip <Foglight_home> TAR: tar -cvf Foglight.tar <Foglight_home> TAR.GZ: tar -zcvf Foglight.tar.gz <Foglight_home> TIP: To restore the database, copy the compressed file to the desired directory and extract this file to the installation directory. If you want to restore the database to a non-embedded PostgreSQL server, see the “Foglight Two Tier” section followed in this table.
Foglight Two Tier
	Foglight Management Server on Windows with MS SQL Server on a remote computer
Option 1			• Use MS SQL tools to back up the database. TIP: To restore the database after choosing this backup method, use MS SQL tools. • Stop the Foglight Management Server. • Use backup tools from your Windows environment to back up the Foglight installation directory.
	Foglight Management Server on Windows with MySQL on a remote computer
Option 1			• Issue a mysqldump command to export the MySQL database. TIP: To restore the database after choosing this backup method, use MySQL. • Stop the Foglight Management Server. • Use backup tools from your Windows environment to back up the Foglight installation directory.
	Foglight Management Server on Windows with PostgreSQL on a remote computer
Option 1			• Stop the Foglight Management Server. • Issue a pg_dump command to export the PostgreSQL database. TIP: To restore the database after choosing this backup method, use PostgreSQL. • Use the root account to dump the Foglight datase. The following is the example of commands: The default password is “foglight” that was set during the installation. The database information can be found in the server.database.* settings under <Foglight_home>/config/server.config. pg_dump.exe -Fc -d foglight -p 15432 -U root -W > <backup_path>/foglightDB.dmp • Use backup tools from your Windows environment to back up the Foglight installation directory.
	Foglight Management Server on Unix with MySQL on a remote computer
Option 1			• Issue a mysqldump command to export the MySQL database. TIP: To restore the database after choosing this backup method, use MySQL. • Stop the Foglight Management Server. • Create a TAR or a ZIP file containing the Foglight installation directory and copy it to a remote drive or a backup tape. TIP: The ZIP format is recommended. TAR files can result in corrupted binary and MySQL data files.
	Foglight Management Server on Windows with Oracle on a remote computer
Option 1			• Use Oracle tools to back up the database. TIP: To restore the database after choosing this backup method, use Oracle tools. • Stop the Foglight Management Server. • Use backup tools from your Windows environment to back up the Foglight installation directory.
	Foglight Management Server on Unix with Oracle on a remote computer
Option 1			• Use Oracle tools to back up the database. TIP: To restore the database after choosing this backup method, use Oracle tools. • Stop the Foglight Management Server. • Create a TAR or a ZIP file containing the Foglight installation directory and copy it to a remote drive or a backup tape. TIP: The ZIP format is recommended. TAR files can result in corrupted binary and MySQL data files.
	Foglight Management Server on Unix with PostgreSQL on a remote computer
Option 1			• Stop the Foglight Management Server. • Issue a pg_dump command to export the PostgreSQL database. TIP: To restore the database after choosing this backup method, use PostgreSQL. • Use the root account to dump the Foglight datase. The following is the example of commands: The default password is “foglight” that was set during the installation. The database information can be found in the server.database.* settings under <Foglight_home>/config/server.config. pg_dump -Fc -d foglight -p 15432 -U root -W > <backup_path>/foglightDB.dmp • Create a TAR or a ZIP file containing the Foglight installation directory and copy it to a remote drive or a backup tape. TIP: The ZIP format is recommended. TAR files can result in corrupted binary and PostgreSQL data files.

The procedures below contain detailed information on how to back up the Foglight installation directory and the database, without backing up the file system or the system registry (on Windows).

IMPORTANT: If you are using an external database and intend to upgrade Foglight after backing it up, you should leave the database running. Doing so allows the installer to successfully upgrade the database.

NOTE: Failing to shut down the Foglight Management Server correctly may result in errors.

Table 11. Where to find more information on different backup methods.

Database	Type	For more information, see..
MS SQL	External	To back up an MS SQL database used by the Foglight Management Server:
MySQL	External	To back up a MySQL database using MySQL commands:
MySQL	Internal	To back up a MySQL database using MySQL commands: or To back up an embedded MySQL database using the InnoDB Hot Backup tool:
Oracle	External	To back up an Oracle database used by the Foglight Management Server:
PostgreSQL	External	Backup and Restore

NOTE: Failing to shut down the Foglight Management Server correctly may result in errors.

Table 12. Where to find more information on different backup methods.

Database	Type	For more information, see..
MS SQL	External	To back up an MS SQL database used by the Foglight Management Server:
MySQL	External	To back up a MySQL database using MySQL commands:
MySQL	Internal	To back up a MySQL database using MySQL commands: or To back up an embedded MySQL database using the InnoDB Hot Backup tool:
Oracle	External	To back up an Oracle database used by the Foglight Management Server:
PostgreSQL	External	Backup and Restore

NOTE: The archive must include the config, cartridge, support, license, and scripts directories and all their content.

TIP: To ensure that the SQL Server remembers your password the next time you attempt to log in, select the Remember password check box.

TIP: For additional information about these settings, see your MS SQL documentation.

TIP: The Foglight database, represented by the foglight node in the Object Explorer tree, is installed on the MS SQL Server during the Foglight Management Server installation. For additional information, see the Installation and Setup Guide set.

TIP: The Backup Up Database—foglight dialog box includes two pages, each showing a different collection of settings: General and Options. By default, the General page is selected. To switch between the General and Options pages, select the appropriate page in the Select a page pane in the upper-left.

TIP: You will need this information when restoring the database from the backup file. For information on how to restore an MS SQL database, see Restoring Foglight.

IMPORTANT: For internal (embedded) MySQL databases only, continue with steps 3 and 4. For external MySQL databases, the backup is now complete.

NOTE: Foglight ships with the InnoDB Hot Backup® tool. You can use this tool to back up an embedded MySQL database as an alternative to the above procedure. For information on how to use this tool, see To back up an embedded MySQL database using the InnoDB Hot Backup tool:.

IMPORTANT: Use the back slash character ‘\’ as a directory separator on Windows platforms; on Unix platforms, use the forward slash ‘/’.

NOTE: The above example illustrates the process of backing up a MySQL database on Windows, which uses the back slash character as a directory separator. On Unix platforms, use the forward slash to separate directories.

NOTE: The above example illustrates the process of backing up a MySQL database on Windows, which uses the back slash character as a directory separator and the dir command to list files. On Unix platforms, use the forward slash to separate directories and the list -l command.

NOTE: The example below illustrates the process of backing up a MySQL database on Windows, which uses the back slash character as a directory separator. On Unix platforms, use the forward slash to separate directories.

NOTE: The code line above illustrates the process of applying logs to an uncompressed backup. If you are applying logs to a compressed backup, use the --uncompress option. For example: C:\Quest\Foglight\mysql\bin>ibbackup --apply-log --uncompress C:\Quest\Foglight\backup \config\backup.cnf For more information about the command-line options for creating compressed and uncompressed backups, see Step 4.

NOTE: The example bellow illustrates the process of backing up a MySQL database on Windows, which uses the back slash character as a directory separator and the dir command to list files. On Unix platforms, use the forward slash to separate directories and the list -l command.

“Restoring” a physical backup means reconstructing it and making it available to users. This section outlines how to restore an old Foglight installation. The process is similar for Windows and Unix systems.

NOTE: To restore a previous Foglight installation, you must have a backed up copy of that version. For information about how to back up a Foglight system, see Backing up Foglight.

Table 13. Select the procedure that matches the backed up database type you have.

Database	Type	Backed up using..	For more information, see..
MS SQL	External	MS SQL commands	To restore an MS SQL database from the database export file:
MySQL	External	MySQL commands	To restore a MySQL database from the database export file:
MySQL	Internal	MySQL commands	To restore a MySQL database from the database export file:
MySQL	Internal	InnoDB Hot Backup	To restore an embedded MySQL database previously backed up using the InnoDB tools:
Oracle	External	Oracle tools	To restore an Oracle database from the database export file:
PostgreSQL	Internal	Zip tools	To restore the previous Foglight installation directory:
PostgreSQL	Internal	Zip tools	To restore the previous Foglight installation directory:
PostgreSQL	External	PostgreSQL commands	To restore a PostgreSQL database on Windows from the database export file:
PostgreSQL	External	PostgreSQL commands	To restore a PostgreSQL database on Unix from the database export file:

TIP: To ensure that the SQL Server remembers your password the next time you attempt to log in, select the Remember password check box.

TIP: For additional information about these settings, see your MS SQL documentation.

TIP: The Foglight database, represented by the foglight node in the Object Explorer tree, is installed on the MS SQL Server during the Foglight Management Server installation. For additional information, see the Installation and Setup Guide set.

TIP: The default location and name of this file is C:\Program Files\Microsoft SQL Server\MSSQL.1\MSSQL\Backup\foglight.bak.

IMPORTANT: In cases where there are multiple databases, it is important to specify which database this import is associated with by issuing the use <database> command. Failing to do so results in an error.

NOTE: The SQL file is the backup file you must create in order to restore your MySQL database (see To back up a MySQL database using MySQL commands:).

NOTE: The files oracle_drop_dp.sql and oracle_drop_db.sql are included with Foglight when you use an Oracle database, The files are located in the <Foglight_home>/scripts/sql directory.

NOTE: The dbdump file is the backup file you must create in order to restore your Oracle database (see To back up an Oracle database used by the Foglight Management Server:).

TIP: Change the pg_hba.conf file to ensure that authentication method is password, if needed. For more information about PostgreSQL connections and authentication, see Connections and Authentication.

NOTE: The database name, username, and password of the backup database and new database must be identical.

NOTE: Make sure to run Foglight and restore the database using the non-root user.

TIP: Change the pg_hba.conf file to ensure that authentication method is password, if needed. For more information about PostgreSQL connections and authentication, see Connections and Authentication.

NOTE: The database name, username, and password of the backup database and new database must be identical.

NOTE: To restore a previous Foglight installation, you must have a backed up copy of that version. For information about how to back up a Foglight system, see Backing up Foglight.

The previous Foglight installation is now restored.

In some cases it may be required to move the Foglight Management Server to different hardware. This may result in a change to the Management Server host name, IP address, or both, while the database may or may not move.

If the host name changes, you must obtain a new Foglight license.To receive a new license, send an email to license@quest.com with your contact and site ID information, along with an explanation of why you need a license. Changing only the IP address, while the host name remains the same, does not require a license update.

If the host name changes, you must also update the CATALYST_URL registry variable. This registry variable contains the Management Server host name and port, which are used to build the alarm URLs sent in email message alerts. For more information about email settings, see Explore the Mail (Global Settings) view.

This sectiontopic describes the process of moving the Foglight Management Server to a different computer.

If you have other configuration changes in your environment that are related to this change, you may need to perform additional configuration steps. Those changes can include:

For more information on how to implement these configuration changes, visit the Support Portal at https://support.quest.com/.

 

Online-Only Topics

Learn more about:

Manage Rules

This dashboard is a starting point for viewing and customizing Foglight rules. It displays a list of all multiple-severity that exist in your environment, including the rules that come with the Management Server and any installed cartridges.

By default, the following columns are displayed:

The above columns appear by default. Click the Customizer to select the following columns for display:

Use the Enable and Disable buttons in the top-left to enable or disable selected rules.

To filter the list of rules by cartridge, click Cartridge in the top left and select a cartridge from the list that appears.

To access the Manage Rules dashboard, click Old Manage Rules. Use this dashboard to access an extended set of rule management tasks that cannot be initiated from the Rule Management dashboard, such as editing rule permissions, and suspending or resuming rule actions.

Foglight has a way to control access to different entities, including rules. While it is possible to use permissions to control access to rules, this feature is deprecated and should not be used.

Copying a rule is useful in situations when you need to quickly create a modified version of an existing rule. If you need to edit an existing rule, it is considered good practice to copy the rule, disable the original rule, then make your modifications. This makes it possible to refer back to the original rule definition, or re-enable it after disabling the modified version, if necessary.

Rules can be copied using either the Rule Management dashboard, or the Manage Rules dashboard. Rules can be deleted using the Manage Rules dashboard.

NOTE: Some browser versions do not support the nested rendering of the Edit Rule page that you are about to access. If you are using those browser versions, click Launch in the display area, as prompted, to open the Edit Rule page in a new window or tab.

NOTE: Some browser versions do not support the nested rendering of the Edit Rule page that you are about to access. If you are using those browser versions, click Launch in the display area, as prompted, to open the Edit Rule page in a new window or tab.

NOTE: Some browser versions do not support the nested rendering of the Edit Rule page that you are about to access. If you are using those browser versions, click Launch in the display area, as prompted, to open the Edit Rule page in a new window or tab.

In some cases, you may need to enable or disable a rule. For example, if a rule monitors a host that needs to taken offline for system maintenance, you can disable that rule temporarily to avoid triggering its actions while the monitored host is unavailable. In most cases this is done using blackouts, however in some circumstances it may be better to disable the rule completely.

Rules can be enabled and disabled using the Rule Management or the Manage Rules dashboard.

TIP: To select multiple rules, select the check boxes in the rows containing the rules that you want to disable.

NOTE: Some browser versions do not support the nested rendering of the Edit Rule page that you are about to access. If you are using those browser versions, click Launch in the display area, as prompted, to open the Edit Rule page in a new window or tab.

TIP: To select multiple rules, press the CTRL or SHIFT key while selecting the rows containing the rules that you want to disable.

TIP: To select multiple rules, select the check boxes in the rows containing the rules that you want to enable.

NOTE: Some browser versions do not support the nested rendering of the Edit Rule page that you are about to access. If you are using those browser versions, click Launch in the display area, as prompted, to open the Edit Rule page in a new window or tab.

NOTE: A Rule is currently disabled icon () appears in the row containing a rule that has been disabled.

TIP: To select multiple rules, press the CTRL or SHIFT key while selecting the rows containing the rules that you want to enable.

You can configure a rule to stop generating alarms for a specified length of time (beginning immediately). It can be useful to suspend alarms in many situations, such as when one or more servers are being brought offline for system maintenance. Rule alarms can be suspended or resumed using the Manage Rules dashboard.

NOTE: While rule alarms are suspended, Foglight stops both generating and clearing alarms for that rule.

NOTE: Some browser versions do not support the nested rendering of the Edit Rule page that you are about to access. If you are using those browser versions, click Launch in the display area, as prompted, to open the Edit Rule page in a new window or tab.

NOTE: The warning icon indicates different types of states a rule may be in. For example, it appears when a rule is disabled, or when its alarms or actions are suspended. Placing the mouse pointer over the icon shows more details about the rule state.

NOTE: Some browser versions do not support the nested rendering of the Edit Rule page that you are about to access. If you are using those browser versions, click Launch in the display area, as prompted, to open the Edit Rule page in a new window or tab.

NOTE: The warning icon indicates different types of states a rule may be in. For example, it appears when a rule is disabled, or when its alarms or actions are suspended. Placing the mouse pointer over the icon shows more details about the rule state.

You can configure a rule to stop performing actions for a specified length of time (beginning immediately). It can be useful to suspend alarms in many situations, such as when one or more servers are being brought offline for system maintenance. Rule actions can be suspended or resumed using the Manage Rules dashboard.

NOTE: Some browser versions do not support the nested rendering of the Edit Rule page that you are about to access. If you are using those browser versions, click Launch in the display area, as prompted, to open the Edit Rule page in a new window or tab.

NOTE: The warning icon indicates different types of states a rule may be in. For example, it appears when a rule is disabled, or when its alarms or actions are suspended. Placing the mouse pointer over the icon shows more details about the rule state.

NOTE: Some browser versions do not support the nested rendering of the Edit Rule page that you are about to access. If you are using those browser versions, click Launch in the display area, as prompted, to open the Edit Rule page in a new window or tab.

NOTE: The warning icon indicates different types of states a rule may be in. For example, it appears when a rule is disabled, or when its alarms or actions are suspended. Placing the mouse pointer over the icon shows more details about the rule state.

The Edit Rule view includes a summary pane that allows you to quickly review a rule’s settings and drill down to the appropriate tab if required. Rule summaries can be accessed using the old Manage Rules dashboard.

The Rule Summary pane includes the following information:

NOTE: Some browser versions do not support the nested rendering of the Edit Rule page that you are about to access. If you are using those browser versions, click Launch in the display area, as prompted, to open the Edit Rule page in a new window or tab.

NOTE: The appearance of the Rule Summary pane depends on the rule type, its severity levels, and other settings. In the above illustration, the rule whose settings appear in the Rule Summary pane is active.

The Rule Diagnostics dashboard lists all rules that exist in your environment, including the rules available with the server and any installed cartridges. Use this dashboard to better understand how the existing rules are operating and to debug any rule-related problems.

For each rule, the Rule Diagnostics dashboard shows the following information:

From here, you can drill down to a rule to see additional diagnostics about that rule. To do that, click the row containing the specific rule and, in the dwell that appears, under Go to, click Diagnostic Details.

TIP: To edit rule definitions, in the dwell, click Edit Rule and edit the rule as required. For more information, see Getting started: create a new rule.

For more information about the Diagnostic Details view that appears, see View diagnostic details for a rule.

The Diagnostic Details view contains detailed run-time information about a selected rule. Use this view to determine which objects are evaluated by the rule and as a debugging tool to help you understand any rule-related problems. Drill down to this view from the Rule Diagnostics dashboard.

This composite view contains the following embedded views:

Table 1. Rule Definition View

Description	Shows rule definitions.
Data displayed	Overview
	• Name. The rule name.
	• Type. A rule type can be either simple or multiple-severity. A simple rule has a single condition, and when that condition evaluates to True, the rule enters the Fire state. Multiple-severity rules are more complex in that they can have multiple conditions and multiple the severity levels, including Warning, Critical, and Fatal. For more information, see Rule types.
	• Rule Scope. The rule scope. This can give you a better idea on the effect the rule has on your monitored system. The scope identifies one or more topology objects that the rule evaluates. If the scope is not defined, the rule runs against the entire data set which can have a negative effect on overall performance. A rule can be scoped to a topology type and can optionally be scoped to specific topology objects of that type. For more information, see Rule scope.
	• Cartridge. The name of the cartridge in which the rule is defined, including the cartridges included with the server, and any installed cartridges. This column is empty for those rules that you create.
	• Cartridge Version. The version of the cartridge in which the rule is defined, including cartridges included with the server, and any installed cartridges. This column is empty for those rules that you create.
	• Comments/Description. Comments or notes associated with the rule.
	• Alarm Description. Description of the alarm message.
	Status
	• Enabled. Indicates if a rule is enabled or disabled. When a rule is disabled, its conditions are not being evaluated which prevents the rule alarms from being generated. For example, if Foglight is collecting any data that, under normal circumstances, result in alarm generation. Failing to generate alarms can be an indication that the rule is disabled. Rules are often brought offline during system maintenance periods.
	• Blackout. Indicates if there are any blackouts in place that affect the rule’s actions.
	• Alarms. Indicates if the rule alarms are enabled or disabled. When rule alarms are suspended, the rule continues to evaluate the collected data but does not generate alarms when its conditions are met. For more information, see Suspending or resuming rule alarms.
	• Actions. Indicates if the rule actions are enabled or disabled. When rule actions are suspended, the rule continues to evaluate the collected data but does not generate actions when its conditions are met. For more information, see Suspending or resuming rule actions.
	Triggering and Behavior
	• Trigger Type. The trigger type. This value affects the frequency at which the rule conditions are evaluated. The rule trigger gives you a general idea of the rule activity and the alarms it generates. For example, a default data-driven trigger causes the rule conditions to be evaluated every time the data associated with the rule is collected and higher data sampling frequencies result in a higher number of alarms being generated. For more information, see Rule triggers.

Table 2. Rulettes for Rule View

Description	Lists one or more object instances to which the rule is bound, and shows information about the rule activity, as it relates to each object instance. Selecting a table row shows additional statistics about the rule activity in the Rulette Details View.
Data displayed	• Long Name. The name of the object instance evaluated by the rule and its data type.
	• Last Reset. The date and time of the most recent reset of the object’s state.
	• Triggers. The total number of times the rule conditions evaluated the object instance.
	• Fired. The total number of times the rule changed the state.
	• Hit. The total number of times the rule conditions evaluated to True.
	• Miss. The total number of times the rule conditions evaluated to False.
	• Last Fired. The most recent date and time the rule changed the state.
	• Last Hit. The most recent date and time a rule condition evaluated to True.
	• Last Miss. The most recent date and time a rule condition evaluated to False.

Table 3. Rulette Details View

Description	For the object instance selected in Rulettes for Rule View, this view shows detailed statistics about the rule execution, broken into severity levels.
Data displayed	Severities
	• Severity. The severity level associated with the condition that was evaluated against the selected object instance.
	• Fire. The total number of times the rule entered the severity level.
	• Hit. The total number of times the condition associated with this severity level evaluated to True.
	• Miss. The total number of times the condition associated with this severity level evaluated to False, or were not evaluated. If a condition is not defined for a severity level, it is not evaluated.
	• Last State. The result of the most recent severity-specific condition evaluation. It can have one of the following values: Hit: The condition evaluated to True. Miss: The condition evaluated to False. Not evaluated: The condition was not evaluated. If a condition is not defined for a severity level, it is not evaluated.
	Observations
	• Name. The name of the metric evaluated.
	• Object Name. The name of the object instance to which the rule is bound.
	• Object Type. The type of the object instance to which the rule is bound.
Where to go next	• Drill down on an entry in the Observations table. Displays a dwell that shows the health of the object instance the metric is associated with, identifies any agents or hosts related to this instance, and lists links to views that reference the object instance, including the Property Viewer and Metric Analyzer. From here, clicking a link under Related drills down to the selected view.

Some Foglight dashboards have reports associated with them. This allows you to run a report based on the current dashboard. You can generate the report using the Reports menu in the top-right corner.

The Rule Management dashboard is associated with the Rules Report. Run this report by choosing Rules Report from the Reports menu, and specifying the input parameters in the report wizard.

The report wizard provides more information about the Rules Report and instructions on how to set the input values. For more information about reports in Foglight, see the Foglight User Help.

Some Foglight dashboards have reports associated with them. This allows you to run a report based on the current dashboard. You can generate the report using the Reports menu in the top-right corner.

The Rule Diagnostics dashboard is associated with the Rules and Registry Variables Report. Run this report by choosing Rules and Registry Variables Report from the Reports menu, and specifying the input parameters in the report wizard.

The report wizard provides more information about the Rules and Registry Variables Report and instructions on how to set the input values. For more information about reports in Foglight, see the Foglight User Help.

Foglight uses flexible rules to apply your business logic to complex, interrelated data from multiple sources within your distributed system. A rule is a piece of business logic that links a condition with a result. It includes a scope and one or more conditional expressions, alarm messages, and actions.

A typical installation can include a large number of rules. You can create and manage multiple-severity rules using the Rule Management dashboard. The Rule Management dashboard lists the rules that exist in your environment and allows you to manage them.

Use this dashboard to drilldown on a rule and to view and edit its definitions. When you edit existing rules allows you can customize how Foglight notifies you of the status of desired metrics and specify what actions should be performed when their status changes.

NOTE: Some browser versions do not support the nested rendering of the Edit Rule page that you are about to access. If you are using those browser versions, click Launch in the display area, as prompted, to open the Edit Rule page in a new window or tab.

TIP: Click a value to view more information about the registry variable and to edit the value.

NOTE: Some browser versions do not support the nested rendering of the Edit Rule page that you are about to access. If you are using those browser versions, click Launch in the display area, as prompted, to open the Edit Rule page in a new window or tab.

Foglight uses flexible rules to apply your business logic to complex, interrelated data from multiple sources within your distributed system. A rule is a piece of business logic that links a condition with a result. It includes a scope and one or more conditional expressions, alarm messages, and actions.

Many rules come included with Foglight and installed cartridges, including Agent Health State, BSM All Events, Catalyst Data Service Discarding Data, and many others. If the existing rules do not meet your needs, you can create a new one and add it to the rule collection.

By creating new rules or editing existing ones, you affect how Foglight notifies you of the status of desired metrics and specify what actions should be performed when their status changes.

Rule definitions specify the time at which they are evaluated, their complexity, and in some cases, the topology objects they evaluate. Rule definitions can be divided into the following categories:

There are two types of rules in Foglight. Those types are Simple rules and Multiple-severity rules.

A simple rule has a single condition, and can be in one of three states:

If its condition is met, the state of the rule is set to Fire and any actions that are associated with this state are performed. If the condition is not met, the rule remains in the Normal state. If the rule’s condition cannot be evaluated because data is missing or unavailable, the state of the rule is set to Undefined and any actions that are associated with this state are performed.

The condition for a simple rule is regularly evaluated against monitoring data. Therefore, the state of the rule can change if the data changes. For example, if the collected data matches a simple rule’s condition, the rule enters the Fire state. If the next set does not match the condition, the rule exits the Fire state and enters the Normal state. You can configure a simple rule to perform one or more actions upon entering and/or exiting each state.

A multiple-severity rule is a more complex type of rule that can have up to five severity levels:

When you create a multiple-severity rule, you must specify a condition for at least one severity level (Fatal, Critical, or Warning).

As with simple rules, the conditions for a multiple-severity rule are regularly evaluated against monitoring data. All conditions in a rule are evaluated; the severity state is set to the highest level for which the condition evaluates to True. If none of the conditions are met, the severity state is set to Normal. If a condition cannot be evaluated because data is missing or unavailable, the state is set to Undefined.

An alarm is generated each time a multiple-severity rule enters a new state. In addition, you can configure a multiple-severity rule to perform one or more actions upon entering and/or exiting each state.

Generated alarms can be viewed in Foglight dashboards that exist outside of the Administration module, such as the Alarms or Agents dashboards. In addition to viewing alarm details, these dashboards allow you to drill down to alarm details, and to clear and acknowledge alarms.

The alarm list can show multiple alarms that are generated by the same rule. This is because rule conditions are evaluated repeatedly, after each data sampling interval. If rule conditions are met in subsequent sampling intervals, Foglight generates an alarm instance for each positive condition evaluation. These alarms can be of the same or different severity, depending on the value of the collected data. For example, a rule can produce multiple Warning alarms, or alarms whose severity ranges from Warning through Critical to Fatal.

Each alarm can be acknowledged or cleared. Acknowledging an alarm indicates that the Foglight operator is aware of the alarm. You can acknowledge a single alarm instance (by clicking Acknowledge in the alarm drill-down dialog box), or all consecutive alarms generated by the same rule (by clicking Acknowledge Until Normal).

Clearing an alarm indicates that the alarm is examined by the Foglight operator and that it is safe to remove it from the list. Along with cleared alarms, any acknowledged alarms can be removed from the list if required using the alarm filter.

For more information about working with alarms, see the Foglight User Guide.

Rule triggers determine the time at which its conditions are evaluated against the collected data. A rule can have one of the following triggers:

Unlike simple rules, that fire without raising any alarms, multiple-severity rules can generate alarms when their conditions are met. These alarms appear in many different dashboard. You can also create your own dashboards that display alarms. In order for alarms to work correctly in dashboards, they must be generated by multiple-severity, data-driven rules. Alarms generated by multiple-severity, time- or event- driven rules do not have the same functionality in dashboards as multiple-severity, data-driven rules.

A time-driven trigger causes one or more of a rule’s conditions to be evaluated once per pre-defined interval. By default, time-driven rules are only evaluated if data for the evaluation of the condition is available, but you can override this behavior, if required.

If a rule has a data-driven trigger, one or more if its conditions will be evaluated every time that new data associated with one or more topology types or objects to which the rule applies is sent to the Foglight Management Server. This option is selected as the default trigger.

An event-driven trigger causes one or more of a rule’s conditions to be evaluated as a response to a certain event. There are four types of events that can act as rule triggers:

A schedule-driven trigger causes one or more of a rule’s conditions to be evaluated during which a selected schedule is in effect. By default, schedule-driven rules are only evaluated if the data for the evaluation of the rule condition is available, but you can override this behavior, if required.

The scope of a rule defines the set of topology objects against which it runs. A rule can be scoped to a topology type and can optionally be scoped to specific topology objects of that type. If a rule is not scoped to specific objects, it applies to all instances of that type. The scope object is the object on which alarms will appear in the Foglight interface. The rule scope is specified using the query language. For more information, see Using the Query Language.

A rule can apply to a topology type or to one or more objects of that type. You can change the scope of a rule (the topology type or one or more specific topology objects to which it applies) after its creation.

Use the Rule Definition tab to edit rule definitions. This tab allows you configure basic settings, such as the rule name, description, and alarm message. It also contains the settings for specifying the rule complexity, its trigger, and scope.

NOTE: This procedure continues from Getting started: create a new rule or Getting started: view and edit rule definitions.

IMPORTANT: The following rule names are reserved and should not be used: foglight_rule_name foglight_rule_comments foglight_rule_domain_query foglight_rule_id foglight_monitored_host_name foglight_monitoring_agent_name foglight_rule_alarm_link foglight_scoping_id. The foglight_monitored_host_name and foglight_monitoring_agent_name variables are only available for rules with scoping queries.

NOTE: These four event types are part of the core Management Server rules. Other event types may appear depending on which cartridges you have installed on the Management Server.

Rule expressions can reference different types of variables:

Foglight registry variables can be used in rule conditions, derived metric expressions, and rule-specific actions. A registry variable can have a global value that is available to all topology types and objects. It can also have one or more values associated with specific topology types or objects, or calendar dates. For more information, see Working with Foglight Registry Variables.

Foglight includes trigger-specific rule variables that can be used in conditional expressions of rules with certain trigger types. Each variable contains the information relative to the rule in which it is used. For example, if you create Rule A and use the rule-level variable foglight_rule_name as a parameter in an action that you add to Rule A, that parameter uses the actual rule name, Rule A.

Rules in Foglight can be triggered by data, time, or events. Different trigger types, such as time, data, and event triggers, have different rule-level variables available to them. For example, in an event-driven rule you can reference the properties of the system alarm event that triggers the rule directly.

Most of the rule variable are available in all contexts within a rule. However, foglight_rule_alarm_link, foglight_monitored_host_name, foglight_monitoring_agent_name and foglight_severity_message are not available in rule condition scripts. Also, foglight_severity_message is also not available at rule level. It is available only as severity level.

Table 4. All rule-level variables and their availability with different trigger types.

Variable	Description	Trigger Type
		Event	Data	Time
args	A collection of arguments provided to the script.  Unused and empty in rule condition/message evaluation.	Yes	Yes	Yes
cartridgeName	The name of the cartridge this rule is deployed in, if available	Yes	Yes	Yes
cartridgeVersion	The version of the cartridge this rule is deployed in, if available	Yes	Yes	Yes
currentEvalTime	The time at which the script is executing.  For data driven rules this is the latest timestamp in the incoming batch of observations.  For time-driven rules this is the time at which the rule fired.	Yes	Yes	Yes
event	System event.	Yes	No	No
event_foglight_is_transition	Indicates if the alarm is transitioning from the Warning, Critical, or Fatal state to Warning, Critical, or Fatal (as applicable). For more information, see isTransition.	Yes	No	No
event_foglight_next_severity_level	The target severity level the alarm is transitioning to. For more information, see nextSeverityLevel.	Yes	No	No
event_foglight_previous_severity_level	The previous severity level the alarm is transitioning from. For more information, see previousSeverityLevel.	Yes	No	No
event_foglight_rule_alarm_link	Link for the alarm causing the event	Yes	No	No
event_foglight_rule_comments	The comments for the rule causing the event	Yes	No	No
event_foglight_rule_id	The ID of the rule causing the event	Yes	No	No
event_foglight_rule_name	The name of the rule causing the event	Yes	No	No
event_foglight_severity_level	The severity level (0-4) causing the event	Yes	No	No
event_foglight_severity_level_name	Name of the severity level causing the event	Yes	No	No
event_scope	Event scope	Yes	No	No
extensionContext	An extension registry used to access interfaces provided by other cartridges	Yes	Yes	Yes
foglight_rule_alarm_link	Link for the alarm	Yes	Yes	Yes
foglight_rule_comments	Comments for the rule	Yes	Yes	Yes
foglight_rule_domain_query	Rule domain query	Yes	Yes	Yes
foglight_rule_id	Rule ID	Yes	Yes	Yes
foglight_monitoring_agent_name	Name of the agent monitoring the scoping topology object. Not all cartridges provide the data for this variable.	Yes	Yes	Yes
foglight_rule_name	Rule name	Yes	Yes	Yes
foglight_scoping_id	ID of the topology object	Yes	Yes	Yes
foglight_severity_level	Severity. Value is an integer in the 0-4 range (inclusive)	Yes	Yes	Yes
foglight_severity_level_name	Name of the severity level	Yes	Yes	Yes
foglight_severity_message	Message on the severity level is used to generate alarms	Yes	Yes	No
rule	Current Rule object	Yes	Yes	Yes
rulette_data	Rulette level persistence map. This data is preserved between rule invocations, but is lost on server restart	Yes	Yes	Yes
scope	The scoping topology object, if one has been set.	Yes	No	Yes
server	A reference to the server and its public API	Yes	No	Yes

NOTE: Foglight prevents you from creating a rule variable with the same name as a system variable. However, the variable name can change if you change the trigger type. In this instance, a warning icon will appear in the Rule Variables and Severity Level Variables listings.

Unlike severity-level variables, that can only be referenced in the expression associated with the severity level in which the variables are defined, rule-level variables can be referenced in any severity level.

There are two types of rule-level and severity-level variables:

Table 5. Examples of expressions and messages in rule- or severity-level variables.

Rule
Name	Value	Type
var1	scope.get("agent/host/name")	Expression
var2	#CPU_Utilization#	Expression
var3	#Run_Queue_Length#	Expression
Text	@var1: CPU Utilization	Message
Subject	CPU Utilization is at @var2% and the number of process in the run queue is @var3. A CPU Bottleneck is being detected on @var1. Check the top processes (using the Top_CPU_Table) to determine which processes are the greatest contributors to CPU Loads, or follow the Foglight online help to find out if the system is CPU constrained. Please use the following URL to obtain alarm details. @foglight_rule_alarm_link	Message

Rule-level variables can be referenced in conditional expressions in multiple severities. When your threshold requirements change, you can create new or edit the existing rule-level variables. Editing an existing rule-level variable affects the outcome of any rule conditions or alarm messages that reference that variable. If you plan to write or edit a conditional expression or an alarm message that includes one or more rule-level variables, ensure that those variables exist prior to writing the expression.

NOTE: This procedure continues from Getting started: view and edit rule definitions or Getting started: create a new rule.

IMPORTANT: The following names are reserved and should not be used: foglight_severity_level and foglight_severity_level_name.

Table 6. Examples of expressions and messages using registry variables.

Severity-Level Variables
Name	Value	Type
var1	scope.get("agent/host/name")	Expression
Text	@var1: CPU Utilization	Message

Severity-level variables can be referenced in conditional expressions associated with the severity level in which the expression is defined. When your threshold requirements change, you can create new or edit the existing severity-level variables. Editing an existing severity-level variable affects the outcome of any rule conditions or alarm messages that reference that variable. If you plan to write or edit a conditional expression or an alarm message that includes one or more severity-level variables, ensure that those variables exist prior to writing the expression.

NOTE: This procedure continues from Getting started: create a new rule.

IMPORTANT: The following names are reserved and should not be used: foglight_severity_level and foglight_severity_level_name.

Table 7. Examples of expressions and messages using registry variables.

Severity-Level Variables
Name	Value	Type
var1	scope.get("agent/host/name")	Expression
Text	@var1: CPU Utilization	Message

A condition is the part of a rule that is evaluated against monitoring data. When it evaluates to True, the rule is said to fire, causing any actions that are associated with the rule or severity level to be performed.

When you create a simple rule, you specify a single condition for it. You can edit this condition after you create the rule. When you create a multiple-severity rule, you must specify a condition for one or more of its severity levels, Fatal, Critical, and Warning, along with an alarm message that is associated with each condition.

A conditional expression can be either true or false. Conditional expressions can reference registry, Groovy functions, and metrics associated with the one or more topology types or their topology objects to which the rule is scoped.

Additionally, conditional expressions can reference properties of topology objects that are related (within the hierarchy of the topology model) to one or more topology objects to which the rule is scoped. For example, the condition for a simple rule that is associated with a specific JVM can reference properties of the server on which the JVM is running (such as the server name), or properties of the cluster to which the server belongs.

Furthermore, event-driven rules can retrieve data generated by report- and alarm-related events.

Expressions can be simple—for example, an expression can consist only of a metric name—but they can also be defined using very complex syntax.

Conditional expressions make use of the query language. For more information, see Using the Query Language.

Foglight generates an alarm message when the conditional expression associated with a multiple-severity rule evaluates to True. An alarm message is typically a text string that can include other severity-level variables, displaying dynamically-supplied data about your monitored system.

Simple rules do not generate alarms. They fire when the condition for their Fire state is met. Multiple-severity rules generate alarms each time they enter a severity state.

Expressions and messages can be set with one of two distinct scopes: Rule-scoped expressions and messages and Severity-scoped expressions and messages.

They can be referenced by the actions set for the Fire and Undefined states of a simple rule and for all severity levels (Fatal, Critical, Warning, Normal, and Undefined) in a multiple-severity rule.

NOTE: It is a best practice that rule-scoped expressions are used until there is a need for more finely scoped expressions.

They can only be referenced by the actions set for the specific rule level at which the message is defined. For example, if an expression is defined for the Fatal level of a multiple-severity rule, it can only be referenced by the actions that are set for that severity level.

A conditional expression is the part of a rule that is evaluated against monitoring data. When it evaluates to True, the rule is said to fire, causing any actions that are associated with the rule or severity level to be performed. A conditional expression can reference variables, topology object metrics, and Groovy functions. Use the Condition tab to specify a conditional expression.

NOTE: This procedure continues from Getting started: view and edit rule definitions or Getting started: create a new rule.

IMPORTANT: Conditional expressions that reference metric properties using the syntax scope.get("aMetric") or metrics using the service layer (for example, server["DataService"].retrieveLatestValue(…)) prevent data-driven rules from firing. Similarly, failing to include a metric reference prevents data-driven rules from triggering. To reference a metric directly use the "#aMetric#" syntax.

IMPORTANT: If you change the metrics, registry variables, or topology object properties that are referenced in the condition, ensure that the new items are available for the topology type or one or more objects to which the rule is scoped.

Table 8. Examples of multi-line and single-line expressions.

Multi-line expressions
	Example A
		Correct
			println @event.dump(); @event.get("report/name") == "MyReport";
		Incorrect
			println @event.dump() @event.get("report/name") == "MyReport";
		NOTE: The first line is missing a semicolon which causes the expression to result in an error.
Single-line expressions
	Example B
		Correct
			println @event.dump();
	Example C
		Correct
			println @event.dump()
NOTE: Using a semicolon to end the line in a single-line expression is optional, therefore both expressions are valid.

IMPORTANT: Do not clear the Activate check box if you want to temporarily disable a rule. To temporarily deactivate the alarms and actions for an entire rule, follow the instructions in Suspending or resuming rule alarms. You can also configure the behavior of the alarms and actions for the rule. See Configure alarm and action behaviorfor more information.

When you write conditions for event-driven rules, in addition to variables, topology object metrics, and Groovy functions, you can use events and their properties to trigger rule actions.

Event-driven rules allow you to monitor the events generated every time a pre-defined event occurs. There are several types of events that can act as rule triggers: AlarmSystemEvent, DatabaseMaintenanceEvent, IncidentSystemEvent, and ReportGeneratedEvent.

Multiple-severity rules generate system events when a rule severity level is reached. Foglight creates an AlarmSystemEvent object instance for each alarm system event. The types of rule severity include Undefined, Normal, Warning, Critical, or Fatal state. The change property of an AlarmSystemEvent object instance indicates the rule severity. Alarm-based system events allow you to trigger alarm-related rules. This can be done by evaluating the values of AlarmSystemEvent properties in rule conditions.

Table 9. The AlarmSystemEvent object properties, their data types, and descriptions

Property	Data Type	Description
alarmID	String	Contains the ID of the alarm that generates the event.
alarmLink	String	Contains the URL to the alarm.
clearedTime	Date	Specifies the time at which the alarm is cleared.
change	AlarmChangeType	Specifies the alarm change type: Fire, Clear, Acknowledge or UserDefinedData.
createdTime	Date	Specifies the time at which the alarm is created.
isAcknowledged	Boolean	Determines if the event is acknowledged. It can be set to True or False.
isCleared	Boolean	Determines if the event is cleared. It can be set to True or False.
isTransition	Boolean	Indicates if the alarm is transitioning from the Warning, Critical, or Fatal state to Warning, Critical, or Fatal (as applicable). For new and clearing alarms, this value is always False. Foglight creates events for newly-created alarms, cleared alarms, or existing alarms transitioning to a new state, as follows: Clearing alarms generate a single Clear event with isTransition set to False. New alarms generate a single Fire event with isTransition also set to False. Existing alarms transitioning from the Warning, Critical, or Fatal state to Warning, Critical, or Fatal (as applicable) generate two events: a Clear event for the old severity, and a Fire event for the new severity. The isTransition flag in both events is set to True. External systems can make use of this behavior when evaluating events, to distinguish between clearing and non-clearing alarms.
nextSeverityLevel	Integer	Indicates the target the alarm is transitioning to. For clearing alarms, the next severity level is always 0 (Normal). Foglight creates events for newly-created alarms, cleared alarms, or existing alarms transitioning to a new state, as follows: Clearing alarms generate a single Clear event with the next severity level of 0 (Normal). New alarms generate a single Fire event with the next severity level of 2 (Warning), 3 (Critical), or 4 (Fatal). Existing alarms transitioning from the Warning, Critical, or Fatal state to Warning, Critical, or Fatal generate two events: a Clear event for the old severity, and a Fire event for the new severity. The next severity level in both events is set to the target severity (Warning, Critical, or Fatal). External systems can make use of this behavior when evaluating Clear events, to distinguish between clearing and non-clearing alarms.
message	String	Contains the alarm message.
previousSeverityLevel	Integer	Indicates the previous severityLevel the alarm is transitioning from. For new alarms, the previous severity level is always 0 (Normal). Foglight creates events for newly-created alarms, cleared alarms, or existing alarms transitioning to a new state, as follows: Clearing alarms generate a single Clear event with the previous severity level of 2 (Warning), 3 (Critical), or 4 (Fatal). New alarms generate a single Fire event the previous severity level of 0 (Normal). Existing alarms transitioning from the Warning, Critical, or Fatal state to Warning, Critical, or Fatal (as applicable) generate two events: a Clear event for the old severity, and a Fire event for the new severity. The previous severity level in both events is set to the original severity (Warning, Critical, or Fatal). External systems can make use of this behavior when evaluating Fire events, to distinguish between clearing and non-clearing alarms.
ruleComments	String	Contains any comments associated with the rule that generates the alarm.
ruleID	String	Contains the ID of the rule that generated the alarm.
ruleName	String	Contains the name of the rule that generated the alarm.
severityLevel	Integer	Contains a number that identifies the severity level: 0: Normal 1: Fire 2: Warning 3: Critical 4: Fatal
severityName	String	Contains one of the following values that identify the severity level: Undefined, Normal, Warning, Critical, or Fatal.
sourceID	String	Contains the ID of the entity that generated the alarm. If the alarm is triggered by a rule, this is the rule ID.
sourceName	String	Contains the name of the entity that generated the alarm. If the alarm is triggered by a rule, this is the rule name.
topologyObjectID	String	Contains the ID of the scoped topology object. If the alarm is triggered by a rule, this is the ID of the topology object to which the rule is scoped.
userDefinedData	DataObject	Contains a data object that includes any additional information about the alarm. This data can be used when creating event-related dashboards. For more information about creating dashboards, see the Foglight User Guide.

Each AlarmSystemEvent object instance contains an AlarmChangeType object instance that stores information about the incident type. You can reference AlarmChangeType property values in rule conditions. For example:

Table 10. The AlarmType object properties, their values, and descriptions.

Property Name	Property Value	Description
Fire	0	Indicates that the alarm is in the Fire state
Clear	1	Indicates that the alarm has been cleared
Acknowledge	2	Indicates that the alarm is acknowledged
UserDefinedData	3	Indicates that the alarm is in a user-defined state

Foglight creates a DatabaseMaintenanceEvent as it performs maintenance operations during the nightly maintenance window. An event will be sent before each operation is run, and after it ends. The operationName property in the event specifies the operation that is being performed. The server also sends a started event for the "DatabaseMaintenance" operation before it runs the individual maintenance operations, and sends a ended event for "DatabaseMaintenance" operation after all the individual operations have been run.

Table 11. The DatabaseMaintenanceEvent object properties, their data types, and descriptions.

Property	Data Type	Description
createdTime	Date	The time that the event was fired.
eventType	DatabaseMaintenanceEvent	An enumeration object that specifies whether this is a “Started” or “Ended” event.
startTime	Date	The time that the operation started.

Table 12. Properties available only for Ended events.

Property	Data Type	Description
endTime	Date	The time that the operation ended.
completed	Boolean	A flag indicating whether the operation completed. An operation will not have completed if it failed or stopped due to time constraints.
errorMessage	String	If a task fails, this property contains the associated error message.

Foglight creates an incident system event when an incident is created or modified. Incidents are used to group alarms based on a common parameter, and can result in ticket requests generated on an external system. For each incident, Foglight creates an IncidentSystemEvent object instance. The types of incident modifications include Create, Close, Acknowledge, Modify, and Delete. The type of incident modification is indicated in the IncidentSystemEvent object’s change property. Incident-based system events allow you to trigger incident-related rules. This can be done by evaluating the values of IncidentSystemEvent properties in rule conditions.

Table 13. The IncidentSystemEvent object properties, their data types, and descriptions.

Property	Data Type	Description
acknowledgedBy	String	Contains the name of the Foglight user who acknowledged the incident.
acknowledgedTime	Date	Contains the date and time at which the incident was acknowledged.
affectedObjects	TopologyObject	Contains the topology object that is affected by the incident.
change	IncidentChangeType	Indicates the type of change made to the incident. There are several types of changes: Create, Close, Acknowledge, Modify, and Delete. For more information about these change types, see their descriptions in the table below.
createdBy	String	If the incident is created manually, as opposed to being generated by the system, this property contains the name of the Foglight user who created the incident.
endedBy	String	If the incident is created manually, as opposed to being generated by the system, this property contains the name of the Foglight user who ended the incident.
endTime	Date	Contains the actual date and time at which the incident ended. If an operator overrides the system-reported end time, this property contains the end time override, stored in endTimeOverride. If no end time override takes place, it contains the system-reported end time, stored in endTimeReported.
endTimeOverride	Date	If an operator overrides the system-reported end time, this property contains the end time override.
endTimeReported	Date	Contains the date and time at which the incident ended, as reported by the system.
incidentID	String	Contains the incident ID.
isAcknowledged	Boolean	Determines whether the incident is acknowledged (True) or not (False).
isEnded	Boolean	Determines whether the incident has ended (True) or is still taking place (False).
lastUpdated	Date	Contains the date and time at which the incident was last updated.
lastUpdatedBy	String	Contains the name of the Foglight user who most recently updated the incident.
linkedAlarmIDs	String	Contains one or more IDs of the alarms that are grouped by the incident.
message	String	Contains a message associated with the incident.
parentID	String	Identifies the parent node in the incident object tree.
problemTicketIDs	String	Contains the ticket ID associated with the incident.
severity	Integer	Contains a number that identifies the severity level: 0: Undefined 1: Normal 2: Warning 3: Critical 4: Fatal
startTime	Date	Contains the actual date and time at which the incident started. If an operator overrides the system-reported start time, this property contains the start time override, stored in startTimeOverride. If no start time override takes place, it contains the system-reported start time, stored in startTimeReported.
startTimeOverride	Date	If an operator overrides the system-reported start time, this property contains the start time override.
startTimeReported	Date	Contains the date and time at which the incident started, as reported by the system.
type	String	Contains the incident type.

Each IncidentSystemEvent object instance contains an IncidentChangeType object instance that stores information about the incident type. You can reference IncidentChangeType property values in rule conditions. For example:

Table 14. The IncidentChangeType object properties, their values, and descriptions.

Property Name	Property Value	Description
Create	0	Indicates that the incident is created
Close	1	Indicates that the incident is closed
Acknowledge	2	Indicates that the incident is acknowledged
Modify	3	Indicates that the incident is modified
Delete	4	Indicates that the incident is deleted

Report generation creates system events. Foglight creates a ReportGeneratedEvent object instance for each report generation event. The type of report properties that you can evaluate include name, reportId, usertemplateName, and many others. Report properties are contained in the ReportGeneratedEvent object’s report property. Report-based system events allow you to trigger report-related rules. This can be done by evaluating the values of ReportGeneratedEvent properties in rule conditions.

Table 15. The ReportGeneratedEvent object properties, their data types, and descriptions.

Property	Data Type	Description
createdTime	Date	Specifies the time at which the event is created.
report	Report	Contains an object of the Report type. For complete information about creating and scheduling reports in Foglight, see the Foglight User Guide.

Each ReportGeneratedEvent object instance contains a Report object instance that stores information about a report. You can reference Report object properties in rule conditions. The following table lists the

Table 16. The Report object properties, their data types, and descriptions.

Property	Data Type	Description
dateRun	Date	The date on which the report is run.
emailRecipients	String	The email recipients to which the report is to be sent.
errorMessage	String	An error message associated with the report.
isEnabled	Boolean	Specifies whether the report generation is enabled.
name	String	The report name.
numResultsRetained	Integer	The number of records in that are retained in the report.
reportId	String	The report ID.
scheduleId	String	The ID of the schedule that is associated with the report.
scheduleName	String	The name of the schedule that is associated with the report.
size	Integer	The report size in bytes.
templateId	String	The ID of the template used to create the report.
templateName	String	The name of the template used to create the report.
user	String	The name of the Foglight user who created the report.

A conditional expression is the part of a rule that is evaluated against monitoring data. When it evaluates to True, the rule is said to fire, causing any actions that are associated with the rule or severity level to be performed. In addition to variables, topology object metrics, and Groovy functions, a conditional expression associated with an event-driven rule can also reference the properties of the event that acts as the rule trigger. For example, you can write a condition for an event-driven rule that fires when Foglight generates a report.

NOTE: This procedure continues from Getting started: view and edit rule definitions or Getting started: create a new rule.

TIP: For a complete list of event properties and the information they contain, see the property definitions in AlarmSystemEvent, DatabaseMaintenanceEvent, IncidentSystemEvent, and ReportGeneratedEvent.

IMPORTANT: The data type of the value you specify here must match the data type of the property with which you compare this value.

IMPORTANT: To separate multiple lines in conditional expressions, use a semicolon followed by a carriage return. In conditional expressions consisting of a single line, using a semicolon to end the line is optional. For more information, see Examples of multi-line and single-line expressions..

IMPORTANT: Do not clear the Activate check box if you want to temporarily disable a multiple-severity rule. To temporarily deactivate the alarms and actions for an entire rule, follow the instructions in Suspending or resuming rule alarms. You can also configure the behavior of the alarms and actions for the rule. See Configure alarm and action behavior for more information.

In some cases you may need to copy the conditions from an existing severity level. A condition is comprised of a conditional expression and an alarm message, both of which can be copied.

Copying a condition can be useful in situations when the conditional expressions of different severities are similar, so instead of writing and validating them for each severity level you can copy an existing expression and modify it to suit your needs.

While you are editing the rule, any unsaved changes to the conditional expressions or alarm messages that you want to copy will be carried over to the destination condition. For example, if you edit a conditional expression for the rule’s Warning severity without saving it, and then proceed to copy that condition in the Critical pane, any unsaved edits of the Warning condition will be carried over to the Critical condition.

Copying a conditional expression from one severity to another without modifying it results in multiple conditions with the same expression. While this type of rule configuration is allowed, a confirmation message appears to make you aware of that. When such a condition is met, it triggers the higher severity. For example, if both Warning and Critical conditions have the same expressions, when that condition is met, the rule enters the Critical severity. An exception to this case is when the expressions resolve the same condition using a slightly different combination of characters. For example, abc>1, abc >1, and abc > 1 are considered different expressions, and as such are evaluated separately.

NOTE: This procedure continues from Getting started: view and edit rule definitions or Getting started: create a new rule.

IMPORTANT: You can only copy the conditions from those severity levels that are already defined. Attempting to copy a condition for a rule that has no conditions defined results in an error.

IMPORTANT: Attempting to copy a condition for a rule that has no conditions defined results in an error.

An action is a particular operation that is performed when a rule enters or exits a state when its condition is met. Simple and multiple-severity rules can be associated with multiple actions. Additionally, each severity level in a multiple-severity rule can have one or more actions. Actions can be added to a rule after it is created.

For example, you can associate an email action with a rule condition. When the rule condition evaluates to True, Foglight sends an email to the selected recipient, with the alarm message in the email body.

There are two types of actions in Foglight:

The actions available in Foglight are as follows:

IMPORTANT: Before you associate a trap action with a rule, you need to ensure that trap actions are configured in Foglight. For details about configuring trap actions, see Configure trap actions. IMPORTANT: SNMP trap actions do not work with data-driven rules.

NOTE: This procedure continues from Getting started: view and edit rule definitions or Getting started: create a new rule.

IMPORTANT: If you choose to associate an SNMP trap action with rule, ensure that trap actions are already configured. For more information, see Configure trap actions.

From here, you can edit the action parameters as required. For more information, see Editing rule action parameters.

Each rule action includes a set of action parameters that can define its behavior. Some action parameters are mandatory while others are optional.

When you add an action to a rule, you must configure the action’s mandatory parameters to ensure its execution when the rule reaches the severity level for which the action is defined. Action parameters can contain Foglight registry variables, rule-level variables, or any custom value. The data type of the specified value must match the action parameter’s data type.

For example, a command action must have the command name associated with it, and in some cases, one or more environment variables. The command name is a mandatory parameter, while the environment variables are optional parameters. However, failing to specify environment variables related the command prevents the command action from being executed.

Table 17. Parameters associated with action types.

Parameter	Required?	Description
BSM Actions
Alarm system event	Yes	The alarm system event generated by Foglight. Syntax: <alarm event> Example: [alarmEvent]/Rule/System Variable Setting this event to the alarmEvent Rule System variable selects all alarms.
alarm.notification.service.backToNormal	Yes	Indicates whether or not send a email notification when the service is back to normal. The recipient will be the same as the one defined in registry variable mail.alias.alarmRecipients.
alarm.notification.service.backToNormal.body	Yes	The body message of the alarm that will be sent when the service is back to normal. The serviceName variable of the service that triggers this alarm will be overwritten.
alarm.notification.service.backToNormal.subject	Yes	The subject of the alarm that will be sent when the service is back to normal.The serviceName variable of the service that triggers this alarm will be overwritten.
BSM URL	Yes	The URL of the BSM system to which to send the data. Syntax: <BSM_URL> Example: http://www.example.com
Event Attributes Managed Control Attributes Technology Level Agreement Attributes	No	Event Attributes, Managed Control Attributes, and Technology Level Agreement Attributes are optional attributes to pass to the BSM system. The BSM action automatically uses the alarm attributes to populate these values, however, it is possible to overwrite them using these parameters. For more information about these attributes, see the Service Discovery and Dashboards documentation. Syntax: <attribute_1>=<value_1>!<attribute_2>=<value_2>!…!<attribute_n>=<value_n> Example: Service=Performance!Value=90
Command Actions
COMMAND_LINE	Yes	The native command that you want to run on the local system where the Management Server is installed. Syntax: <command> [arg_1] … [arg_n] Example: C:\notepad.exe This command starts the Notepad executable on the C drive. If the command is not accessible from the <Foglight_home> directory, you need to specify its path.
ENVIRONMENT_VARIABLES	No	A list of environment variables that you want to reference on the command line. Multiple environment variables can be separated by exclamation marks ‘!’. Syntax: <var_1>=<value_1>!<var_2>=<value_2>!…! <var_n>=<value_n> Example: MY_PROFILE=C:\Users\jsmith!MY_SHARE=\\example.com\home\jsmith
Email Actions
mail.attachement	No	The name of the file to be added as an attachment to the email. Example: Report1.pdf
mail.attachement.file.name	Yes	The file name of the email attachment that you want to send in the email. Syntax: <file.extension> Example: test.docx
mail.attachement.mime.type	Yes	The MIME type of the email attachment, specified using the <application>/<file type> format. Syntax: <application>/<file type> Example: Acrobat/pdf Supported MIME types do not depend on the Management Server settings, but on the email server.
mail.bcc	No	A list of email addresses to send as a Blind CC list. Uses a comma ‘,’ as a separator. Syntax: <email_address_1>,<email_address_2>,…,<email_address_n> Example: jsmith@example.com,ldoe@example.com
mail.cc	No	A list of email addresses to send as a CC list. Uses a comma ‘,’ as a separator. Syntax: <email_address_1>,<email_address_2>,…,<email_address_n> Example: jsmith@example.com,ldoe@example.com
mail.content.type	Yes	This parameter can be set to text/plain or text/html. Syntax: <text/plain>|<text/html> Example: text/html
mail.message	No	The text of the email message. Syntax: <email_text> Example: This is an email message from Foglight.
NOTE: When the condition of a simple rule is met, the rule enters the Fire state. By default, simple rules do not generate alarms when their conditions are met. Some simple rules that ship with Foglight create observation alarms that are not directly associated with the original rule. This is accomplished by calling the checkObservationAlarms() function in the rule condition. When the condition of a simple rule includes a call to the checkObservationAlarms() function, and the rule includes an email action, in the message body of the resulting email, the following text always precedes the value specified by the mail.message parameter: The following alarms have been raised. Please use the provided URLs to obtain alarm details. For more information about the checkObservationAlarms() function, see Using Functions in Conditions and Expressions.
mail.recipient	Yes	The email address of the recipient to whom you want to send the email message. It overrides the value set by the global mail.recipient registry variable. Syntax: <email_address> Example: jsmith@example.com
mail.subject	No	The subject line of the email message that you want to send. Syntax: <email_subject> Example: Email message from Foglight.
Remote Command Actions
COMMAND_LINE	Yes	The command that you want to execute on the monitored system. Syntax: <command> [arg_1] … [arg_n] Example: C:\notepad.exe This command starts the Notepad executable on the C drive of the monitored system. If the command is not accessible from the <Foglight_home> directory, you need to specify its path.
ENVIRONMENT_VARIABLES	No	A list of environment variables that you want to reference on the command line. Multiple environment variables can be separated by exclamation marks ‘!’. Syntax: <var_1>=<value_1>!<var_2>=<value_2>!…! <var_n>=<value_n> Example: MY_PROFILE=C:\Users\jsmith!MY_SHARE=\\example.com\home\jsmith
HostName	Yes	The name of the host computer on which the command is to be executed. Syntax: <host_name> Example: Host1
MatchAll	No	When set to true, the command is executed on all hosts matching the selection criteria. When set to false, the command executes only on the first matching host. Syntax: <true|false> Example: true
PlatformInfo	No	The target platform specification. Syntax: <OS_name> <OS_version> <OS_architecture> Example: windows 5.0 ia32
RemoteInstallationId	No	The installation ID of the Agent Manager. This information is useful if there are multiple remote agents that support remote command execution. Syntax: <Agent Manager installation ID> Example: 6624671a-e5b5-4bd5-9278-4e5ecc1ff173 This ID can be found in the <Agent_Manager_home>\state\default\config\fglam.config.xml file, between the <config:id> tags.
RemoteWorkingDir	No	Working directory on the remote machine. Syntax: <directory_path> Example: C:\Users\jsmith
UseRegExp	No	Indicates whether the values specified by the HostName, PlatformInfo, and RemoteInstallationId parameters are regular expressions. Syntax: <true|false> Example: true
Script Actions
Argument 1-10	No	Since the script action executes a custom script, the meaning of these arguments depends on the script requirements. For example, if the script requires a host name and an agent name in that order, then Argument 1 should contain the host name and Argument 2 should be the agent name. Syntax: <Argument> Example: Host1.example.com
Scoping object id	No	The ID of the scoping object. Syntax: <topology_object_ID> Example: 01db-dc768-36ad-48b4-d33c-76bdc3145cbd
Script name	Yes	The name of the script. Syntax: <script_name> Example: myScript
NOTE: Any scripts you reference here must be imported into the server as a persistable script using a monitoring policy import action. Only then the script can be selected for use with a script action. The script has to be wrapped in an XML structure that uses the following syntax: <?xml version="1.0" encoding="UTF-8"?> <foglight-config version="1.0.2.0" exported-on="2010-08-31T19:50:57.344Z"> <persistable-script> <script_name>RecoverDeadAgents</script_name><![CDATA[ YOUR SCRIPT HERE script-meta-information> <script-property name="script_description"><![CDATA[Recover Dead Uncontrolable Agents]]></script-property> </script-meta-information> </persistable-script> </foglight-config> The XML can be imported into the server using the util:configimport command. For more information about this and other fglcmd commands, see the Command-Line Reference Guide.
Send SNMP Trap Actions
CommunityString	Yes	The SNMP community string.
SNMPVersion	Yes	The SNMP version. Syntax: <1|2> Example: 2
TargetAddress	Yes	One or more trap target addresses, comma-separated. Syntax: <host_1>,<host_2>,…,<host_n> Example: myHost
TargetPort	Yes	The target port of the trap receiver. Syntax: <port_number> Example: 162
AgentIDOID	No	The object ID of the agent ID that is defined in the MIB file. Syntax: <object_ID> Example: 1.3.6.1.4.1.7572.1.4.3
AgentIDValue	No	Agent ID value of the trap variable defined in the MIB file.
AgentNameOID	No	The object ID of the agent name that is defined in the MIB file. Syntax: <object_ID> Example: 1.3.6.1.4.1.7572.1.4.4
AgentNameValue	No	Agent name value of the trap variable defined in the MIB file.
AgentTypeOID	No	The object ID of the agent type that is defined in the MIB file. Syntax: <object_ID> Example: 1.3.6.1.4.1.7572.1.4.2
AgentTypeValue	No	Agent type value of the trap variable defined in the MIB file.
AgentMessageOID	No	The object ID of the agent message that is defined in the MIB file. Syntax: <object_ID> Example: 1.3.6.1.4.1.7572.1.4.1
AgentMessageValue	No	Agent message value of the trap variable defined in the MIB file.
DateTimeOID	No	The object ID of the date and time format that is defined in the MIB file. Syntax: <object_ID> Example: 1.3.6.1.4.1.7572.1.4.14
DateTimeValue	No	Date/time value of the trap variable defined in the MIB file.
HostIPOID	No	The object ID of the host IP that is defined in the MIB file. Syntax: <object_ID> Example: 1.3.6.1.4.1.7572.1.4.11
HostIPValue	No	Host IP value of the trap variable defined in the MIB file. Syntax: <IP_address> Example: 168.212.226.204
HostMACOID	No	The object ID of the host MAC address that is defined in the MIB file. Syntax: <object_ID> Example: 1.3.6.1.4.1.7572.1.4.12
HostMACValue	No	Host MAC value of the trap variable defined in the MIB file. Syntax: <MAC_value> Example: 00-0B-C2-78-76-BA
HostnameOID	No	The object ID of the host name that is defined in the MIB file. Syntax: <object_ID> Example: 1.3.6.1.4.1.7572.1.4.10
HostnameValue	No	The host name value of the trap variable defined in the MIB file. Syntax: <host_name> Example: myHost
RuleIDOID	No	The object ID of the rule ID that is defined in the MIB file. Syntax: <object_ID> Example: 1.3.6.1.4.1.7572.1.4.5
RuleIDValue	No	The Rule ID value of the trap variable defined in the MIB file. Syntax: <rule_ID> Example: 630861d6-fa0d-45f0-9a64-47b617df9c5e
RuleNameOID	No	The object ID of the rule name that is defined in the MIB file. Syntax: <object_ID> Example: 1.3.6.1.4.1.7572.1.4.6
RuleNameValue	No	The rule name value associated with the trap in the MIB file. Syntax: <rule_name> Example: Agent Health State
SeverityOID	No	The severity object ID defined in the MIB file. Syntax: <object_ID> Example: 1.3.6.1.4.1.7572.1.4.9
SeverityValue	No	The severity value of the trap variable defined in the MIB file. Syntax: <severity> Example: Warning
TopologyObjectIDOID	No	The object ID of the topology object ID that is defined in the MIB file. Syntax: <object_ID> Example: 1.3.6.1.4.1.7572.1.4.7
TopologyObjectIDValue	No	The topology object ID value of the trap variable defined in the MIB file. Syntax: <topology_object_ID> Example: 01db-dc768-36ad-48b4-d33c-76bdc3145cbd
TopologyObjectNameOID	No	The name of the topology object ID that is defined in the MIB file. Syntax: <object_ID> Example: 1.3.6.1.4.1.7572.1.4.8
TopologyObjectNameValue	No	The topology object name associated with the trap in the MIB file. Syntax: <object_name> Example: Host1.example.com
URLOID	No	The object ID of the URL value defined in the MIB file. Syntax: <object_ID> Example: 1.3.6.1.4.1.7572.1.4.13
URLValue	No	The URL value of the trap variable defined in the MIB file. Syntax: <URL> Example: example.com

The following rules apply to the command syntax for Command and Remote Command actions:

NOTE: This procedure continues from Associate a rule with an action.

IMPORTANT: The Type column shows the parameter’s data type. When changing the parameter value, ensure that the value you specify matches that data type.

IMPORTANT: When specifying the parameter value, ensure that the value you specify matches the parameter’s data type. For more information, see Step 2.

CAUTION: SNMP trap actions do not work with data-driven rules. For more information about different types of rule triggers, see Rule triggers.

In Foglight, Simple Network Management Protocol (SNMP) trap actions can be configured to forward alarms to an SNMP trap receiver. Trap definitions are specified in a Management Information Base (MIB) file. MIB files contain data structures that define the types of events that are being monitored and controlled.

Configuring SNMP trap actions assumes that you plan on sending SNMP traps from your monitoring environment to an external SNMP trap receiver. The process of forwarding SNMP traps is triggered by rules. This means that a rule that has an SNMP trap action assigned to it can cause Foglight to send an SNMP trap to a trap receiver when the rule’s condition is met.

There are several ways that you can use to configure SNMP trap actions. The simplest and probably most convenient way is to edit the Forward Alarms as SNMP Traps rule that is included with a core installation of Foglight.

This rule relies on the definitions of the MIB file, quest-foglight.mib. Both the Forward Alarms as SNMP Traps rule and the quest-foglight.mib file are included in the Send-SNMP-Trap-Action cartridge, deployed during the Foglight Management Server startup.

This type of configuration involves editing the SNMP community string and the trap receiver’s IP address defined in the Forward Alarms as SNMP Traps rule. When configured, the rule sends traps that are formatted according to the settings specified in the SNMP trap action parameters. The rule contains an entering Send SNMP Trap Action with a set of pre-defined parameters that are compatible with the MIB file.

In addition to editing the SNMP community string and the trap receiver’s IP address, it is possible to define advanced SNMP settings, such as the trap target address or variable bindings using this rule. You can either edit, or copy and edit the Forward Alarms as SNMP Traps rule to meet your needs. Another option is to define SNMP traps in new or existing rules by adding an entering or exiting Send SNMP Trap Action to a rule, and edit the action’s parameters. This process typically requires advanced knowledge of the SNMP protocol and is beyond the scope of this guide. For more details, consult your SNMP documentation.

NOTE: Some browser versions do not support the nested rendering of the Edit Rule page that you are about to access. If you are using those browser versions, click Launch in the display area, as prompted, to open the Edit Rule page in a new window or tab.

In some cases you may need to copy the severity-level variables and actions from an existing severity level of the same rule. Each severity level can have its own actions and severity-level variables.

While you edit rules, any unsaved changes to the severity-level variables or actions that you want to copy will be carried over to the destination severity. For example, if you edit an action for the Warning condition of a rule without saving it, and then proceed to copy that action in the Critical pane, the unsaved edits of the Warning action will be carried over to the Critical severity.

Copying severity-level actions and variables can be useful in situations when those definitions are identical or similar. Instead of writing and validating them for each severity level you can copy existing ones and modify them, if required.

NOTE: This procedure continues from Getting started: view and edit rule definitions or Getting started: create a new rule.

IMPORTANT: You can only copy variables and actions from those severity levels that are already defined. Attempting to copy a condition for a rule that has no conditions defined results in an error.

IMPORTANT: Attempting to copy a condition for a rule that has no conditions defined results in an error.

You can associate a rule with an effective schedule or a blackout schedule. An effective period is a schedule during which you want a rule to be evaluated. For example, you might want to set your company’s hours of operation as the effective period for a rule.

You can also set blackout periods for a rule. A blackout is a schedule during which evaluation of the rule is suspended for set intervals. For example, you might want to set the times when regularly scheduled maintenance is performed on a server as the blackout period for a rule.

If a rule has no schedules associated with it, then it is always active. If you only add effective schedules to a rule, then it is automatically inactive at all times other than those specified by the effective schedules. Conversely, if you only add blackout schedules to a rule, then it is automatically active at all times other than those specified by the blackout schedules.

If you add both effective and blackout schedules to a rule, then it will be active only at the times specified by the effective schedules minus the times specified by the blackout schedules. This is because blackout schedules take precedence over effective schedules. For example, suppose you add two schedules to a rule: an effective schedule that runs Monday to Friday, 9 am to 5 pm, and a blackout schedule that runs every Tuesday from 10am to 11am. The rule will be active every Monday, Wednesday, Thursday and Friday from 9 am to 5 pm but will only be active from 9am to 10am and from 11am to 5pm on Tuesdays.

NOTE: This procedure continues from Getting started: view and edit rule definitions or Getting started: create a new rule.

NOTE: Adding the Always entry to the list of blackout schedules does not create a black out for the rule. It has no effect on the rule’s blackout schedule.

Foglight allows you to configure rule behavior to ensure it does not fire repeatedly. Defining the behavior of rule alarms and actions can help you avoid being overwhelmed with alerts when a rule condition is met many times within a short period.

The following options are available:

NOTE: This procedure continues from Getting started: view and edit rule definitions or Getting started: create a new rule.

Table 18. All selected alarm and action behaviors must be true in order for the rule to fire.

Rule condition met at:	Selected alarm and action behavior options	Behavior condition met?		Rule fires?
		A	B
1:00		Yes	No	No
1:15		Yes	No	No
1:20		Yes	No	No
1:50		Yes	No	No
2:00		Yes	Yes	Yes

Foglight allows you to create flexible rules that can be applied to complex, interrelated data from multiple sources within your distributed system. You can associate several different actions with a rule, configure a rule so that it does not fire repeatedly, and associate a rule with schedules to define when it should and should not be evaluated.

Different types of data can be used in rules, including registry variables, raw metrics, derived metrics, and topology object properties.

There are two types of rules in Foglight: simple rules and multiple-severity rules. A simple rule has a single condition, and can be in one of three states: Fire, Undefined, or Normal. A multiple-severity rule can have up to five severity levels: Undefined, Fatal, Critical, Warning, and Normal.

Rule conditions are regularly evaluated against monitoring data (metrics and topology object properties collected from your monitored environment and transformed into a standard format). Therefore, the state of the rule can change if the data changes. For example, if a set of monitoring data matches a simple rule’s condition, the rule enters the Fire state. If the next set does not match the condition, the rule exits the Fire state and enters the Normal state.

A rule condition is a type of expression that can be true or false. When it evaluates to true, the rule is said to fire, causing any actions that are associated with the rule or severity level to be performed. You can configure a rule to perform one or more actions upon entering or exiting each state. When a multiple-severity rule fires, an alarm also appears in Foglight.

For more information see Configure Rules and Metric Calculations to Discover Bottlenecks .

The Foglight Management Server includes some built-in rules that monitor the health of your application server environment. Rules in this section:

NOTE: The Foglight Management Server also includes a rule that can be used to configure SNMP Trap Actions. For more information about this rule, see Forward Alarms as SNMP Traps rule. Foglight includes two kinds of trap actions: SNMP Trap Actions and Send SNMP Trap Actions. Both trap action technologies are currently supported, however the Send SNMP Trap Action is a new version of trap actions, and is recommended by Quest over the use of SNMP Trap Actions. Similar to SNMP Trap Actions, Send SNMP Trap Action can be configured using the Forward Alarms as SNMP Traps rule, described in this topic.

Both trap action technologies are currently supported, however Quest recommends the use of Foglight Trap Actions over SNMP Trap Actions.

This rule monitors the health of all agents in the monitoring environment. It generates an alarm if it finds an agent whose health is deteriorating (Warning) or is down (Critical).

Agent : agentID != "0"

Table 19. Agent Health State Rule Definition

Conditions	State
An agent’s health is in decline	Warning
An agent is down	Critical

This rule sends all alarms from Foglight to the Service Discovery and Dashboards product.

The BSM All Events rule includes an entering BSM action which includes two mandatory action parameters: Alarm system event and BSM URL. These parameters must be set in order for the BSM rule to work properly. By default, Alarm system event is set to the alarmEvent rule-level expression, while the BSM URL action parameter points to the BSM URL Foglight registry variable. To ensure this rule works as expected, you need to configure the BSM URL Foglight registry variable to the destination address, followed by enabling this rule (it is disabled by default).

For more information about the Service Discovery and Dashboards, see the product documentation.

None

This rule monitors the credentials that have the Validity Window policy set. It fires when it finds any credentials that are about to expire.

CatalystServer

Table 20. Catalyst Credential Check Rule Definition

Conditions	State
A credential is currently valid but is to expire in five days or earlier.	Warning
A credential is no longer valid.	Critical

This rule monitors the observations and generates an alarm if the Data Service starts discarding any observations. This can happen when the Foglight Management Server is overloaded, or when there is a difference, or the difference in the system time between the monitored system and the Foglight Management Server. This alarm indicates the server is overloaded and cannot keep up with the incoming data.

CatalystDataService

Table 21. Catalyst Data Service Discarding Data Rule Definition

Conditions	State
The Data Service discards one or more observations within a 15 minute interval	Warning

This rule monitors the size of the database, checking whether the database size is higher than the predefined threshold, set by the DBSMon.MaxDatabaseSize registry variable. By default, this value is set to 100 Gb. If required, you can increase this value.

CatalystDatabase

Table 22. Catalyst Database Space Checking Rule Definition

Conditions	State
The size of the Foglight database exceeds 75% of the maximum database size.	Warning
The size of the Foglight database exceeds 90% of the maximum database size.	Critical
The size of the Foglight database exceeds 100% of the maximum database size.	Fatal

This rule checks the space that is currently available to the Oracle database against the thresholds defined in the Foglight registry. It generates alarms when the Oracle table space becomes too large. The thresholds for generating alarms are set by the following variables:

Database administrators should provide values for these thresholds in order to get notified when the database starts growing out of bounds.

CatalystTablespace

Table 23. Catalyst Free Space Checking for Oracle Tablespaces Rule Definition

Conditions	State
The size that is available to the Oracle database exceeds the threshold set by the DBSMon.WarningFreeTablespaceSize registry variable.	Warning
The size that is available to the Oracle database exceeds the threshold set by the DBSMon.CriticalFreeTablespaceSize registry variable.	Critical
The size that is available to the Oracle database exceeds the threshold set by the DBSMon.FatalFreeTablespaceSize registry variable.	Fatal

This rule periodically clears old LogFilter alarms.

None

This periodically looks for new agent instances that are connecting to the Foglight Management Server. It rebuilds the topology if it detects new agents.

CatalystServer

This rule directs all scheduled reports to their email recipients. A scheduled report can have one or more email recipients.

None

Checks whether the CPU count of an agent type exceeds the licensed number of agents. It generates a Warning alarm if it finds that the number of monitored hosts an agent is currently monitoring is higher than the number allowed by your license.

NOTE: This rule only applies to Foglight servers with a limited license.

AgentTypeLicense

Table 24. Catalyst Agent Type License Checker Rule Definition

Conditions	State
The number of monitored hosts is higher than the number allowed by the license.	Warning

This rule checks the amount of time the Foglight Management Server spends for garbage collection and generates alarms if that time exceeds pre-defined thresholds, defined by a set of registry variables for each severity state: Warning, Critical, and Fatal.

(CatalystServer).jvm.garbageCollectors where name not like '%copy%'

Table 25. Catalyst Garbage Collector Rule Definition

Conditions	State
The amount of time spent on garbage collection exceeds the threshold set by the registry variable FMSMon.gcWarn. The default value of that variable is 10.	Warning
The amount of time spent on garbage collection exceeds the threshold set by the registry variable FMSMon.gcCritical. The default value of that variable is 30.	Critical
The amount of time spent on garbage collection exceeds the threshold set by the registry variable FMSMon.gcFatal. The default value of that variable is 90.	Fatal

This time-driven rule checks the validity of the current licenses, and generates alarms if it finds that a license is about to expire within a certain time period.

CatalystServer

Table 26. Catalyst License Monitoring Rule Definition

Conditions	State
A license expires in the number of days set by the LicenseExpirationDaysWarning registry variable. By default, this number is 30.	Warning
A license expires in the number of days set by the LicenseExpirationDaysCritical registry variable. By default, this number is 7.	Critical
A license expires in the number of days set by the LicenseExpirationDaysFatal registry variable. By default, this number is 2.	Fatal

This rule checks the memory that is available to the Foglight Management Server and generates a Critical alarm if the server is in danger of running out of memory, defined as 95% memory utilization. If this alarm is generated and cleared occasionally, this does not indicate any potential problems, however, if the alarm stays active without clearing, or if it is generated and cleared frequently, this indicates that you need to increase the memory allotment.

(CatalystServer).jvm

Table 27. Catalyst Memory Usage Check Rule Definition

Conditions	State
The amount of memory that is available to the Foglight Management Server exceeds 95% of the total memory usage.	Critical

Checks if any attempts to create topology objects are failing because the topology size limit has been reached. This number is defined by the foglight.limit.instances registry variable whose global default value is set to 10,000. You can change this value if required.

CAUTION: Increasing the default value of the foglight.limit.instances variable may cause performance issues on the Foglight Management Server. If you need to increase this value, contact Quest Support for further instructions.

The setting of this threshold protects against volatile, untuned topology models. This can be often caused by JavaEE Request URL tuning. If this rule fires, in most situations agent tuning is required to make the data less volatile.

CatalystTopologySizeConstraintService

Table 28. Catalyst Topology Size Limit Rule Definition

Conditions	State
The attempts to create topology objects are failing because the maximum number of topology objects exceeds the value set by foglight.limit.instances.	Warning

This is a template rule that can direct all incoming SNMP traps to an SNMP trap receiver, once the rule’s Send SNMP Trap Action parameters, CommunityString and TargetAddress are set to point to the desired SNMP trap receiver.

You can use this rule as a template when creating rules with SNMP trap actions. For more information about viewing the settings related to SNMP trap actions and their configuration in Foglight, see Configure trap actions .

None

This rule periodically checks whether there are any idle agents. An agent is considered idle if it is running but the Foglight Management Server does not register any data associated with that agent for a pre-defined period of time, defined by a registry variable for each severity state: Warning, Critical, and Fatal

Agent

Table 29. Idle Agents Rule Definition

Conditions	State
The agent is idle for the number of hours set by the registry variable IdleAgent.Warning. The default value of that variable is 1.0 hours.	Warning
The agent is idle for a period of time set by the registry variable IdleAgent.Critical. The default value of that variable is 24.0 hours.	Critical
The agent is idle for a period of time set by the registry variable IdleAgent.Fatal. The default value of that variable is 168.0 hours.	Fatal

This rule checks whether at least one instance of the Foglight Agent Manager is running on a monitored host.

Host : detail instanceof RemoteClient

Table 30. Remote Agent Manager State per Host Rule Definition

Conditions	State
No instance of the Foglight Agent Manager on a monitored host.	Warning

Foglight monitors each service (either implicit or user-defined) for service level compliance. The ServiceLevelEvaluation – FMSServiceSLP rule checks the availability of each service and raises an alarm if the availability is lower than the a predefined threshold during a period of one hour.

FSMServiceLevelPolicy

Table 31. ServiceLevelEvaluation - FMSServiceSLP Rule Definition

Conditions	State
An average availability during one hour period is below 95%.	Warning
An average availability during one hour period is below 85%.	Critical
An average availability during one hour period is below 70%.	Fatal