Foglight 6.3.0 - Administration and Configuration Guide

Persistence Handler Overview

The Persistence Handler Overview dashboard contains charts that display information about the persistence handler such as the number of rows in the table, data length, index length, batch insert time, and query time.

Data Management

Use the Object Cleanup dashboard when you need to inspect, purge, and delete data objects, and particularly for removing objects that are no longer needed.

As an example, deleting agent instances from the Services dashboard removes the agent definitions. However, you can still view the agent entries in other dashboards such as the dashboard for Oracle. To remove the agent entries from these dashboards, you need to delete the agents using the Object Cleanup dashboard to remove the services and OracleModel Instances that were created for the Oracle dashboard.

Foglight builds topology models at run-time using the data collected from your monitored systems, and keeps information related to monitored types indefinitely. Global Default setting can be used so objects are not kept indefinitely. Objects that are not expected to change or update are considered stale. These stale objects consume monitoring resources and can be cleaned up. If you have a very large number of stale objects, this may indicate a misconfiguration of your monitoring environment.

Use the Object Cleanup dashboard to remove stale objects, and to reconfigure your URL transformation rules if necessary.

CAUTION: Editing the topology model is an advanced procedure that can significantly impact your Foglight environment. Quest Software strongly recommends that you contact Support for guidance before attempting to use this dashboard.

The following walkthrough demonstrates how to locate and manage stale types based on the default settings provided. In this example, we are using Catalyst Serviceability Object as the type with too many objects.

IMPORTANT: After you type a new number in the since box, you must press Apply to ensure the value is updated.

TIP: The time range includes the last full week/month/year, and the days between the end of that time range and the current date. For example, if today is Thursday, and you select Inactive since 2 Weeks ago, then the time range starts at the third Monday previous, and includes two complete weeks (Monday to Sunday), plus 3 more days.

IMPORTANT: After you type a new number in the since box, you must press Apply to ensure the value is updated.

TIP: The time range includes the last full week/month/year, and the days between the end of that time range and the current date. For example, if today is Thursday, and you select Inactive since 2 Weeks ago, then the time range starts at the third Monday previous, and includes two complete weeks (Monday to Sunday), plus 3 more days.

IMPORTANT: The object is removed from the topology. However, any data collected for the object remains in the database. If the object is recreated (for example, by reinstalling a cartridge), the data becomes available again.

NOTE: In previous releases you could add object types to the list on the dashboard. Beginning with Foglight 6.3.0, the cartridge monitoring policy specifies which types appear in the list.

You can configure the number of days each type object is allowed to remain inactive before being automatically deleted.

Use the Retention Policy dashboard to set system-wide policies for managing database growth through automatic data aggregation and data purges. You can use standard policies (for example, short-term, long-term, or managed roll-ups) or create a custom retention policy.

When a specific retention policy has not been defined, the system-wide retention policies are used by default.

NOTE: The default retention policy is set to purge data after 13 months. To change this setting, create a custom retention policy.

If you find that the standard retention policies do not meet your needs (that is, long-term policy retains granular data for a longer period than what is required) you can create a custom retention policy.

For example, if you want to change the retention policy to have more granular data (that is, for at least one day) such as for capturing reporting data. After a day, you want to roll up that data to an hourly average and kept for a week. Beyond that, you want to keep a daily average for 15 months.

You can also navigate to an advanced dashboard to set rollups per object, using the Manage Retention Policies dashboard in the Administration module. For more information, see Manage Data Retention.

To create a custom retention policy:

Use the Manage Monitored Objects table to look at some of the common objects (organized by domain) in Foglight. For example, you can see that some old host and agent objects should be removed as they were part of the early testing of deployment. For each object you can see the amount of data for that object and its children in the total size. This is a handy breakdown as it gives you an idea of what data is returned.

Table 4. The following categories appear by default in the Monitored Objects table.

Category	Column	Description
Object	Name	Name of the object or its children. Click object to view a popup for additional details about the object.
Direct database size		Estimate of the metric instances for the object and its children.
	Topology	Displays the size of data (accumulated and directly) and its dependent objects.
	Observations	Gives an idea of the direct (topology) size and any non-containment property.
Purged		Shows whether the current node has been purged. If Yes appears in the column, you can click it to see the purge details.
Last Updated		Shows the last time at which data was collected for a particular object and its children. This column ignore model alarm count calculations.

Use the Delete, Purge for objects, or Purge for types options in the header of the table to manage data objects.

Delete — Use this option to immediately delete the object, the object metric, or delete the object tree. You can choose whether to delete an object individually or delete its tree and also choose whether or not to delete the metrics associated with an object. For example, if you choose to delete a host, all data associated with the host is deleted.

Purge for objects — Use this option to schedule the removal of the observations for a selected object during the database off-hours maintenance cycle. By default, this maintenance is scheduled at the end of the month. For more information, see Purge data objects.

Purge for types — Use this option to schedule the removal of objects by types during the database off-hours maintenance cycle. By default, this maintenance is scheduled at the end of the month. For more information, see Purge data objects.

Use the Direct Database Size area to evaluate the current storage allocation for each object in the system and to purge/delete unwanted objects from the system. For example, you can figure out where space is used by viewing how much and where the space is being used.

By viewing the size of the direct database you can see immediately how much space is being consumed by the database, as well as the number of metrics. You can also see a breakdown of space by metric age. For example, from the table you can tell there is a lot of old data and you can decide if you want to keep some old data, or which data to delete or purge.

Use the purge feature if you want to keep the topology object for a nightly cycle. Within a 24 hour period, the data object should be gone from the topology tree.

TIP: You can use the Search feature located at the top of the Manage Monitored Objects table to find the data object such as by using regular expressions.

NOTE: If there is more than one observation purge applied to the same object, the longest purge age will be used. NOTE: All previously defined observation purges for the object are overwritten.

When you delete an object, you actually remove the topology object immediately from the tree, including all of its children. For example, you may want to delete a data object if you have old data on a decommissioned host and want to remove the topology objects. If you delete an object that is still active, it will be re-created.

NOTE: Metrics are always preserved and deleted as part of this topology maintenance operation.

Management Server View and Management Server Metrics

The Management Server View dashboard is useful for examining the server performance. It can help you diagnose potential performance problems and identify bottlenecks in your environment.

Object Groups

Most of the object groups needed for service monitoring already exist. Administrators can use the Object Groups dashboard to classify a group of data and then use the Tier Definition dashboard to subscribe to which data groupings you want to use as tiers. Object groups do not bind to data.

An object group is a mapping to a certain set of data types of the objects you are interested in. Once an object group is defined, its mapping can be evaluated to return these objects. Because the result depends on what exist in Foglight by the time the evaluation is done, its membership list may vary.

Foglight ships with a number of default object groups and subgroups. These groups are marked as Created by Foglight and cannot be changed or deleted.

If an object group is used as a tier, it cannot be deleted. If an object group or object subgroup is used in a rule to define a dynamic group of components in a service, it cannot be deleted.

An object group cannot be used to define a rule in the Service Builder unless it contains more than one subgroup. Conversely, you are not allowed to delete all subgroups from an object group if the object group is used to create a rule to dynamically include a group of components in a service.

After creating an object group, you need to add one or more subgroups to that group. For more information, see Create an object subgroup.

An object subgroup contains one or more objects, selected using a common criteria, that make up that subgroup.For example you can create a subgroup that contains all agent instances whose names start with “tor”.

Use the Tier Definitions dashboard to associate an object group to a tier. A tier is another logical component that references one or more object groups, based on some common criteria. This type of logical structure helps you investigate performance problems associated with multiple objects. For example, to investigate the state of the monitored components within a tier, drill down on a tier and investigate the hosts that are related to it. For more information about services, see the Foglight User Guide.

The object groups that are made available in the Tier Definitions dashboard are based on object groups that were created using the Object Groups dashboard. For more information, see Create an object group.

The Object Groups that you select in the Tier Definitions dashboard appear in the Tier drop down list when selecting a new local or global service using the Service builder dashboard. These tiers are also used in the Service Operations Console, where they show service breakdown by tier.

NOTE: New tiers are hidden by default in the Service Operations Console. To display a new tier, in the Service Operations Console, click Customize Service Operations Console, open the Tier Selector tab, and ensure the newly-created tier is selected. For more information about the Service Operations Console, see the related help page.

The FMS Overview view is a starting point for performance analysis of the Foglight Management Server. It displays information about the database activity, data service performance, JVM performance, and server load in one unified view.

In federated environments, explore the diagnostics for each individual server: Federation Master, Federation Child, or a standalone Foglight Management Server. To do that, click Choose a Foglight Server and select a host from the list that appears.

This view provides more information on:

NOTE: To populate the metrics showing database inserts per 5 minutes for Oracle and SQL Server, you need to grant additional permissions to the database user. For more information, see Grant permissions to explore database activities.

Table 5. The following table describes what type of information to look for in the Performance Overview view.

To find out about…	See this view	Performance implications
Database inserts		Database insert spikes, slow processing
Data handling		Incoming data spikes (batchesProcessed increases), slow processing (batchesDiscarded increases)
Memory utilization		Too large topology, too much incoming data (freeMemory increases)
Server load		Load is higher than 0.7

The Agents view is designed to see what agents you are running, listed by type. By selecting agent types in the tree, a graph showing the number of agents as a function of time is plotted. This graph is handy for viewing changes to the agents, for example if a number of agents that you did not know about were added, or checking if old agents are properly removed.

The Baseline Measurements view shows internal database I/O statistics related to baseline computation. A metric baseline is the expected metric range for the given time period. When configured, IntelliProfile computes baselines for metrics, and stores that data in the database.

Table 6. Look for the following types of information in the Baseline Measurements view.

To find out about…	See this view	Performance implications
Loaded baseline data		High peaks in the graph usually indicate an impact on the database performance, depending on the database capabilities.
Stored baseline data		High peaks in the graph usually indicate an impact on the database performance, depending on the database capabilities.
Server requests to load baseline data		High values in the graph may be a sign that baselining activities have a high impact on the database performance, depending on the database capabilities.
The number of metrics whose baseline data is removed from the database		This graph shows zero values at all times, unless baseline profiles are explicitly removed for trouble-shooting purposes, usually requiring manual intervention.
The number of requests to remove baseline profiles		This graph shows zero values at all times, unless baseline profiles are explicitly removed for trouble-shooting purposes, usually requiring manual intervention.
The number of server attempts to store baseline data		Normally these values should correspond to those appearing in the Store Call Count graph. If they are greater that means that some attempts failed due to database-related errors.
The number of requests to store baseline data		High peaks in the graph usually indicate an impact on the database performance, depending on the database capabilities.
The number of baseline management operations		High peaks in the graph usually indicate an impact on the database performance, depending on the database capabilities.
The number of outstanding baseline management operations		Outstanding baseline management operations can include enabling or disabling baselines for a particular metric. This graph can be expected to show zero values at all times.

The Connectivity view shows the Foglight connectivity for the JDBC connection pool and the user session count. It summarizes the database connection state from the perspective of the Foglight Management Server. By looking at this view, you can determine if there are enough JDBC connections to service application requests.

To find out about…	See this view	Performance implications
JDBC connections		Overused pool indicates a slow database.
User sessions		High number of sessions results in a high browser interface load, indicating a slow browser interface/server.

The Database Overview view displays the metrics that represent the reads, inserts, and updates of the database rows. These metrics can be used in performance load tests to track the unit load.

Contents of this view includes a drop down to select the host name. Here, you can select a Federation Master, Federation Child, or standalone server. To do that, click Choose a Foglight Server and select a host from the list that appears. It also contains the following embedded views:

The name of the metrics that contain database reads, inserts, and updates depend on the type of the supported database that Foglight uses to store information, as indicated in the following table.

Table 8. Database and recording metric types.

Database	Metric
MS SQL, PostgreSQL	num_rows_deleted
	num_rows_inserted
	num_rows_updated
MySQL	innodb_rows_inserted
	innodb_rows_inserted_per5min: #innodb_rows_inserted# * 300
	avg_inserts_per5min: avg(#innodb_rows_inserted_per5min for 60 minutes#)
Oracle	qa_db_inserts : select sum(inserts) from sys.dba_tab_modifications where TABLE_OWNER='xxxx'
	delta_db_inserts: delta(#qa_db_inserts from QA_DB_INSERTS_Agent_Table_DB_INSERT_COUNT#)
	delta_avg_per5min: avg(#delta_db_inserts for 60 minutes#)

NOTE: Permissions for MySQL and PostgreSQL are granted automatically. However, to populate the database dashboard metrics for Oracle and Microsoft SQL Server, special permissions must be granted for the database user in the Foglight Management Server. For more information, see Grant permissions to explore database activities.

Table 9. The following table describes what type of information to look for in the Database Overview view.

To find out about…	See this view	Performance implications
Database row operations		High number of database operations may indicate a possible database bottleneck. The cause could be related to a problem on the server sid, such as topology changes.
Database I/O		High load may indicate a possible database bottleneck. Check the database specifications.
Database buffer		May indicate a dirty database buffer, possibly because the innodb pool buffer is too small.
Database inserts		Check if there is anything more than row operations.

This view shows the invocation counts of and time spent on garbage collection. It contains two embedded views:

The Java Virtual Machine (JVM) view provides full details on the JVM memory activity. This view is particularly useful if you are performing detailed JVM tuning for the different memory spaces.

To find out more about the JVM heap generation, observe the spikes on the embedded views. Big spikes with a slow climb are better than small spikes (too little available memory) or a quick climb (frequent garbage collection).

This view shows the number of rulettes (rule instances) that are bound to specific object instances. This view is useful for understanding the environment complexity.

The Rulette and Topology view shows how many rules and rule instances are running. Many rulettes, especially as a function of the number of objects, can indicate that you are overloading your server or if some old rules are still running.

This view lists Foglight scripts with the script ID, script name, sum (ms), count, the difference between minimum and maximum invocation time, and the name of the cartridge in which the script is defined. A plot summary of the script invocation time also appears.

This view shows the Foglight Management Server load metrics and JVM performance.

This view highlights the data and metrics that are being processed in time or getting discarded.

The Service - Data & Message view shows details on data handling of the Foglight Management Server such as the number of skipped messages and the number of discarded metrics. The top-right embedded view shows the percentage of data that is dropped by the server over time, while the top-left embedded view shows the counts of total and skipped messages over time. A high percentage of dropped data can often indicate a server overload. The embedded view in the bottom right is useful for indicating how long data processing is expected to take. If that value grows over time, that is a good indication that the Foglight Management Server cannot keep up with the load from the agents.

The Service - Derivation and Query view shows activity in the derivation and query service such as the number of evaluations, rulettes, errors, and cache performance.

This view shows a mix of graphs that indicate the cost of metrics such as how many derived metrics are running in the Foglight Management Server, how many metric evaluations have occurred, and information on finding metrics in memory. Metrics are resource-demanding if there are a lot of derived metrics compared to the number of topology objects, or if the memory settings are such that no metric history is being kept in memory.

The Service - Persistence view shows metric details for the Foglight Management Server that is related to:

This view shows the number of topology changes, skipped messages, and total messages. This view is useful for understanding the topology structure relating to agent activity.

The Service - Topology & Agent Manager is a two-part view.

The UI Query Time view is similar to the Script Invocation Time view (see page 205) except that it shows tables for a query instead of a script.

This view shows a query table listing the query name, query ID, amount of time spent on invocation, count, and difference between minimal and maximal invocation time. A graph summarizing query invocation time also appears.

Some Foglight dashboards have reports associated with them. This allows you to run a report based on the current dashboard. You can generate the report using the Reports menu in the top-right corner.

The Performance Overview dashboard and Management Server View are associated with the Management Server Performance Summary Report. Run this report by choosing Management Server Performance Summary Report from the Reports menu, and specifying the input parameters in the report wizard.

The report wizard provides more information about the Management Server Performance Summary Report and instructions on how to set the input values. For more information about reports in Foglight, see the Foglight User Help.