Foglight 5.9.1 - Administration and Configuration Guide

Configure Rules and Metric Calculations to Discover Bottlenecks

The Foglight™ Management Server and individual cartridges each come with a set of topology types, data retention policies, rules, and calculations, that, in most cases, work out-of-the-box. In more complex environments, your business case may require additional tuning.

Foglight uses flexible rules to apply your business logic to complex, interrelated data from multiple sources within your distributed system. A rule is a piece of business logic that links a condition with a result, for example if HTTP response time exceeds 500 ms, fire a warning alarm. A rule can includes a scope and one or more conditional expressions, alarm messages, and actions. The scope defines the set of topology objects against which it runs. The conditional expression defines the thresholds, that, if reached, cause the rule to fire. Conditional expressions can include registry variables, raw metrics, derived metrics, and topology object properties.

There are two types of rules in Foglight: simple rules and multiple-severity rules. Simple rules do not generate alarms, they fire and invoke actions when their conditions are met. Multiple-severity rules include up to five severity levels and generate alarms when the condition associated with any one of its severity levels is met.

Each severity level can have its own set of variables that you can use in alarm messages. Unlike registry variables, which are global in nature, severity-level variables are only accessible to the severity level in which they exist. For example, a Warning-level variable that contains alarm text can only be referenced by the alarm message defined for the Warning severity. Critical- or Fatal-level alarm messages, associated with the same rule, do not have access to this variable.

Severity levels can be associated with actions, causing them to occur each time a threshold is reached. Foglight comes with different types of actions, such as email, command, script, and other types of actions.

Rules have four types of triggers: data-, time-, schedule-, and event-driven triggers.

Many rules come included with Foglight and installed cartridges, such as Agent Health State, BSM All Events, Catalyst Data Service Discarding Data, and others. If the existing rules do not meet your needs, you can create a new one and add it to the rule collection.

A typical installation can include a large number of rules. You can create and manage multiple-severity rules using the Rule Management dashboard. To access this dashboard, on the navigation panel, click Dashboards > Administration > Rules & Notifications > Rules. The Rules dashboard provides access to modifying threshold variables appearing in rule conditions. For an extended set of rule management tasks, such as viewing and editing rule definitions, copying, disabling, and enabling rules, suspending or resuming rule actions, use the Manage Rules dashboard. Click Old Manage Rules to access the Manage Rules dashboard.

To obtain additional diagnostics about rule behavior and how they affect your monitoring environment, use the Rule Diagnostics dashboard. From here, drill down to a specific rule and explore the objects are affected by the rule, or find out how many times a rule was executed against a specific object. This can help you understand rule behavior and debug any problems associated with a particular rule. To access this dashboard, from the navigation panel, click Dashboards > Administration > Rules & Notifications > Rules, then click Diagnostics from the rule’s context menu.

Related topics:

For more information, see the following topics:

Working with Derived Metrics

In addition to building topology models at run-time using the data collected from monitored systems, Foglight has a unique capability to apply pre-defined calculations to the collected metrics. Metric calculations are typically scoped to specific parts of the topology and their values can change over time. They can be reused in expressions to simplify their syntax and speed up system deployment.

A metric is a specific value that is measured over time. There are two types of metrics in Foglight: raw metrics and derived metrics. Raw metrics are collected directly from your monitored environment and sent to the Foglight Management Server. Derived metrics are calculated from one or more raw or derived metrics. They are scoped to a topology type and can optionally be scoped to specific objects of that type. Many derived metrics come included with Foglight and installed cartridges. If none of the existing derived metrics meet your needs, you can create a new one and add it to the derived metric collection.

There are many reasons why it can be useful to create derived metrics. For example, creating derived metrics can make managing rules simpler by reusing metric expressions.

You can create and manage derived metrics using the Manage Derived Metrics dashboard. A typical installation can include a large number of rules. The Manage Derived Metrics dashboard lists all derived metrics that exist in your environment, and allows you to drill down to derived metric definitions. To access this dashboard, from the navigation panel, click Dashboards > Administration > Data > Derived Metrics.

To obtain additional diagnostics about derived metric behavior and how they affect your monitoring environment, use the Derived Metrics Diagnostics dashboard. From here, drill down to a specific derived metric and explore the objects that are affected by the derived metric, or find out how many times a derived metric has been calculated against a specific object. This can help you understand derived metric behavior and debug any problems associated with a particular derived metric. To access this dashboard, in the Administration dashboard, click Metrics Diagnostics in the Data column.

For more information, see the following topics:

Working with Metric Thresholds

Threshold levels in metrics are useful in situations when you need to reference a specific metric value multiple times, for example in derived metrics or rules. Each metric can have one threshold associated with it. A threshold is always associated with a threshold level. Threshold levels refer to a particular state of monitoring entities, such as agent states, alarm severities, and others. Each threshold level includes a unique set of threshold bound levels that are specific to that level. For example, the threshold level AgentState comes with several bound levels that relate to agent states, such as Running and Collecting Data.

Creating a threshold involves selecting a metric and defining values for threshold bounds. A bound level value can be a constant value, a registry variable, or another metric of the same topology type. As data is sampled, Foglight evaluates the metrics for which thresholds are defined, matching their run-time values with bound-specific values in pre-defined order, and performs actions when specific bound levels are reached.

You can create and manage thresholds using the Manage Thresholds dashboard. To access this dashboard, from the navigation panel, click Dashboards > Administration > Data > Manage Thresholds.

For more information, see the following topics:

Analyze Activity Levels with IntelliProfile

Foglight uses the IntelliProfile technology to estimate the system performance and help with system monitoring and planning. IntelliProfile uses data collected during a desired time period and generates a baseline operating range based on the collected metrics.

NOTE: You can only use this feature after installing a baseline cartridge that contains definitions for a selected metric property. Agent developers can write baseline cartridges. For more information about metric thresholds, see Working with Metric Thresholds.

A baseline establishes expected data patterns for a given time period. Baselines are periodically evaluated during IntelliProfile learning cycles, to reflect changes in data patterns.

During operating cycles, IntelliProfile compares incoming data for those metrics that have IntelliProfile threshold levels configured. Metric threshold states reflect the degree of deviation from the baseline, and can indicate potential performance bottlenecks. If there are any rule conditions that evaluate threshold states for such metrics, Foglight can generate alarms when a metric enters a certain threshold state.

IntelliProfile settings can be viewed and managed using the IntelliProfile dashboard. To access this dashboard, from the navigation panel, click Dashboards > Administration > Data > IntelliProfile.

For more information, see the following topics: