Foglight 7.3.0 - Data Model Guide

Where can I see the data model hierarchy? What internal models are created? What other models are created? What is in "Model Roots"? How are Foglight 4 agents represented? Why might I need to navigate models? How do I create a model? How do I delete a model?

What are domains?

Example: the Host Model structure

What is a data source? What are "KnowledgeItems"?

Data modeling tutorials

Modeling example: The org chart

Defining the org structure types Populating the model using a script Connecting "Employees" to Host objects Visualizing the data

Appendix: Groovy scripts

createOrg Groovy script

Appendix: Internal database schema

acl_class Table acl_entry Table acl_object_identity Table acl_sid Table agent_client_defaults Table agent_config_binder Table agent_dc_manager_schedule_ids Table agent_dc_manager_state Table agent_manager_state Table alarm_alarm Table alarm_annotations Table alarm_loc_msg Table auditing_log Table baseline_config Table baseline_config_properties Table baseline_engine_profile Table baseline_observation_profile Table cartridge_cartridge_relation Table cartridge_components Table cartridge_installed_cartridges Table cartridge_items Table credential_data Table credential_lockbox Table credential_mapping Table credential_mapping_entry Table credential_order Table credential_policy Table current_version Table database_instance_id Table database_version Table derivation_calculation Table derivation_complex_definition Table derivation_definition Table fgl4_migration_agent Table fgl4_migration_data_span Table fgl4_migration_dcm Table fgl4_migration_host Table fgl4_migration_host_mapping Table fgl4_migration_log Table fgl4_migration_server Table incident_affected_objects Table incident_incident Table incident_linked_alarms Table incident_problem_ticket Table incident_problem_tickets Table licensing_licenses Table mgmt_object_size Table mgmt_observation_size Table mgmt_timeslice Table mgmt_timeslice_data_avail Table model_association Table model_property_formula Table model_query_criteria Table obs_binary_* Tables obs_metric_aggregate_* Tables obs_metric_scalar_* Tables obs_string_* Tables pcm_encoded_data Table persistable_config_model Table persistable_script Table persistence_column_mapping Table persistence_db_column Table persistence_db_schema Table persistence_db_table Table persistence_grouping_policy Table persistence_lifecycle Table persistence_lifecycle_period Table persistence_obs_key_purge_age Table persistence_obs_purge Table persistence_obs_purge_age Table persistence_observation_index Table persistence_operation Table persistence_retention_policy Table persistence_rollup_progress Table persistence_rollup_retry Table persistence_storage_config_xml Table persistence_storage_manager Table persistence_timeslice_table Table persistence_topobj_purge_age Table persistence_type_hierarchy Table registry_performance_calendar Table registry_registry_value Table registry_registry_variable Table report_output Table report_schedule Table rule_action_handler Table rule_action_message Table rule_action_registry_reference Table rule_action_variable_reference Table rule_blackout_schedules Table rule_effective_schedules Table rule_expression Table rule_firing_strategy Table rule_messages Table rule_rule Table rule_sev_to_clear_actn_hndlr Table rule_sev_to_fire_actn_hndlr Table rule_severity Table rule_severity_expression Table rule_severity_messages Table schedule_named_schedule Table script_annt Table script_annt_attr Table script_argument Table script_argument_annt Table script_argument_annt_attr Table script_example Table script_return_annt Table script_return_annt_attr Table sec_group Table sec_group_nesting Table sec_group_role_match Table sec_grouprole Table sec_jaas_source Table sec_object Table sec_object_mask Table sec_object_permission Table sec_object_type Table sec_permission Table sec_permission_def Table sec_policy Table sec_resource Table sec_role Table sec_user_alias Table sec_user_obj_permission Table sec_user_res_permission Table sec_usergroup Table sec_userrole Table sec_x_attribute Table sec_x_attribute_value Table tagging_service_mapping Table threshold_bound Table threshold_config Table topology_activity_calendar Table topology_activity_upgrade Table topology_object Table topology_object_history Table topology_property Table topology_property_annotation Table topology_property_history Table topology_property_name Table topology_property_value Table topology_service_state Table topology_type Table topology_type_annotation Table topology_type_history Table upgrade_pending_operations Table wcf_groups_by_cartridges Table wcf_resources Table

What are "KnowledgeItems"?

What are “KnowledgeItems”?

KnowledgeItem is a concept that allows users to attribute business-relevant information to their monitored infrastructure. This information may include (but is not limited to) cost-center attributes, business unit associations, or configuration and application management metadata.

The Management Server pre-defines an abstract type KnowledgeItem that contains basic information:

</type>

Concrete instances of sub-types can be used to extend these knowledge items for a specific purpose (for example, business unit information), and are linked to the corresponding monitored topology information via the reference 'asset'.

The Management Server will offer management of these knowledge items in the UI in a later release. At the moment, users should extend the KnowledgeItem type as desired, and create business-related knowledge content and references in scripts.

The following is an example of a simple business-unit related knowledge item:

<types>

</type>

</types>

To add real value to this basic knowledge item, extend this type, by renaming and customizing it appropriately — see <foglight_home>/extension/knowledge-items/acme-types.xml

The types-file can be imported into the model by copy and paste, or as a file upload in the Administration > Data > Add Topology Type dashboard.

After users define various KnowledgeItems, other knowledge items instances can be created programmatically. For example:

// run in script editor, assuming a target scope (the asset) is set

if (scope==null)

return "No scope/asset to attach KI to"

ts = server.TopologyService;

t = ts.getType("AcmeBUKnowledgeItem");

ki = ts.getObjectShell(t);

ki.set("asset", scope);

ki.set("name", "R&D Business Unit Asset #1");

ki.set("description", "Host hosting business unit R&D");

ki.set("lineOfBusiness", "Research");

ki.set("businessUnit", "R&D");

ts.mergeData([ki]);

The Management Server shows the KnowledgeItems in the model under Services > All Model Roots > Knowledge Items and Root of Monitoring > KnowledgeItemModel >knowledgeItems. To see the data model tree, in the navigation panel, under Dashboards, click Configuration > Data.

The server comes with an example that shows how to surface the knowledge items in the UI. The example dashboard depends on a Acme example KnowledgeItem type (see <foglight_home>/extension/knowledge-items/acme-types.xml) that first needs to be imported through the Add Topology Type dashboard. The example dashboards can then be imported by using the following command:

fglcmd -cmd util:uiimport -f acme-dashboards.zip

When accessing the Service Operations Console (in the navigation panel, under Dashboards, click Services > Service Operation Console), all Acme knowledge items for elements inside the selected service are then shown in an extra tab, named ACME BU Knowledge Items.

Data modeling tutorials

This section provides a tutorial that walks you through the process of adding new functionality to Foglight, by building new models and without changing the Foglight core code.

The technique for extending Foglight is shown in the following illustration.

Figure 1. Extending the Foglight model

The process consists of the following steps:

Define your new types.

Create your types through data — Groovy functions or agents.

Create dashboards based on your type instances.

Re-iterate (repeat Step 1 through Step 3) until you get the result you want.


	NOTE: If you make an error in your type definitions, you do not have to throw away your server. You can change your definition on the fly, and Foglight adapts. There are also functions available to remove the data. This means that it is possible to iterate over a model definition and work towards the result you want. You do not have to sit down with a Unified Modeling Language (UML) tool for a day or two.

Modeling example: The org chart

Foglight is not just a monitor, it is an object-oriented application server. Using Foglight’s modeling capabilities, you can extend Foglight to model data from pretty much any domain.

This example walks you through an exercise that extends Foglight to represent a company’s Org structure. We create the model, load it into the server, populate it using a Groovy script, and create some dashboards to visualize the result:

•

Defining the org structure types

•

Populating the model using a script

•

Connecting “Employees” to Host objects

•

Visualizing the data

A company’s org chart shows all the different departments in the company, along with their relationships.

Figure 2. The company org chart

To make this exercise interesting, we are going to model the org chart and the employee reporting hierarchy. Each organization has a leader, and that leader has direct reports. Then, we are going to connect the IT department to the actual IT infrastructure. The result is a meaningful org chart that shows the state of the IT and the root cause of potential outages from an employee perspective.

Figure 3. The company org chart structure

Defining the org structure types

To demonstrate the power of models, we are going to create a company org structure inside Foglight.

The org structure consists of organizations that contain other organizations. The organizations are lead by managers. The managers have direct reports.

We will create a model that represents these relationships, populate the model, then visualize it using dashboards. Then, we will create a relationship between this Org Structure model and Host objects, to show how models can be connected.

Org structure: Type hierarchy and relationship

There are three key types in our Org Structure example: Organization, Manager, and Employee. It seems natural that a Manager should be a specialization of an Employee. It also seems natural that there is a relationship between the organizations, and also between the employees.

As always with anything object-oriented, there are multiple ways to solve the problem of relationships between objects. Here are the key relationships we want to maintain:

•

We want to know what sub-organizations make up different organizations.

•

We want to know who owns each organization.

•

We want to know who works in each organization.

•

We want to know who reports directly and indirectly to each manager.

•

We want to be able to have managers who do not own an organization.

•

We want the contact information for every employee.

To do that, we are going to create three objects: Organization, Manager, and Employee. Manager is a subclass of Employee.


	NOTE: You could also create an extra object, Department, as a subtype for Organization. But for this example, we will keep things simple and create just the previously mentioned objects.

All the relationships we need are going to be modeled using three collections:

•

Each Organization has a Manager defined by Organization.orgLeader.

•

Each Organization has a list of child Organizations defined by Organization.childOrgs.

•

Each Manager has a list of direct reports (Employee objects) defined by Manager.directReports.

Before we get into the type definitions for each of our three types, we need to figure out what defines the identity of each type. In Foglight, identity is critical, because it determines when a new object is created.

In a real-world case, the identity of each employee is defined by an employee ID. For this example, we are going to define the identity of an employee based on his or her first and last name. That feels like a unique value for an individual, at least in a small organization. For Organization, we are going to define the identity based on the name of the organization. The reasons behind choosing this option are explained in the following scenarios:

•

If the identity of an organization included the name of the manager of the organization, what would happen if we changed the organization’s leader? We would have a new organization, and that is not right. The organization should remain unique, even if the manager changes.

•

If the identity of an organization included the child organizations, then we would not be able to create departments without changing the parent organization into a new instance.

•

If the identity of a manager included the organization that he or she manages, then we would not be able to handle promotions cleanly.

Therefore, we are going to use the following identity information for our types (as illustrated in the following diagram):

•

Employee: identity fields lastName and firstName.

•

Manager: same identity as Employee, has a directReports collection of Employees.

•

Organization: identity field orgName. Has an orgLeader link to the organization’s manager, and has a childOrgs list of departments that make up the organization.

Figure 4. Organization structure

Now we can finally get started on creating some types inside Foglight. After designing the object/type hierarchy, getting the types into Foglight is relatively easy.

Foglight model definition for org chart

Rather than examining the syntax for topology type definitions, we are going to study real examples.

In Foglight types, a type can extend another type. Most types extend TopologyObject in order to inherit alarm propagation and roll-up features. Each type contains a set of properties. A property can be:

•

A metric (that is, time-series data like “CPU Utilization”).

•

A Java® type, like “String” or “Integer”.

•

Another object.

•

A collection of other objects.

The “Employee” type

We can get started by looking at the simplified definition for the Employee type. It contains two properties, lastName and firstName:

</type>

To load this type definition:

On the navigation panel, under Dashboards, click Administration > Data > Add Topology Type.

The Add Topology Types dashboard appears.

In the Import From Text box, keep the pre-populated text in the dialog, and insert your type definition between the <types></types> tags.

Figure 5. Add Topology Type dashboard

To validate the type definition, click Validate.

If the code passes the validation, the Successful dialog box appears, confirming the validation result.

Click Close.

To import the new type definition, click Import.

The Successful dialog box appears, confirming the result of importing the new type definition.

Click Close.


	IMPORTANT: You can repeat this procedure as many times as you want. As you modify your type, Foglight adapts. Later on you can add new fields, and they are added to the type definition. This is important to understand. It means you can iterate over a type in a Foglight server without worrying about backing out the older versions.


	NOTE: To review the definitions for all the important types, including TopologyObject and Host, see FGLHOME/config/topology-types.xml. To review the DTD definition for the topology type XML file format, see FGLHOME/dtd/topology-types.dtd.

The Employee type definition is now part of the Foglight topology types. We should highlight the following characteristics:

•

Employee extends TopologyObject. This means that it inherits all the alarm roll-up capabilities that most Foglight objects use. We want this capability because we want to be able to show the state of the systems managed by an Employee.

•

Both lastName and firstName are String properties.

•

Both lastName and firstName are identity properties. This means that the combination of first and last name define a unique Employee object instance. You can have two Employee objects with the same last name, or the same first name. But the combination is considered unique.

Next, we are going to look at a slightly extended version of the Employee object:

</type>

This (partial) definition includes new properties, which hold basic contact information for each employee. We have added jobTitle, phoneNumber, and emailAddress. These properties are a little different than lastName and firstName. Specifically, they are not identity properties. What that means is:

•

An Employee is not uniquely defined by his or her phone number, for example. A change to a phoneNumber property does not create an Employee instance.

•

Foglight automatically tracks changes to non-identity properties as topology changes. This means Foglight notes the time of the change, and the old-new value pair. Therefore, we get the object change tracking for free (for example, we can find out when an employee’s phone number changed).

The next interesting definitions are for ownedItems and interestingItems, which are used to model an employee’s system responsibilities. If Bob in IT is responsible for three machines, we want to be able to represent that relationship. We have chosen to add two collections to represent this relationship:

•

ownedItems: represents those items for which an employee is responsible. The state of these items impact the state of the employee. If Bob in IT has three machines, and one of them is in a Fatal state, then the Employee object representing Bob is in a Fatal state. This is made to happen by setting is-containment to true.

•

interestingItems: represents things that an employee finds interesting, but for which they are not responsible. This allows us to maintain a relationship between an employee and other things that does not result in alarm state rolling up. This is made to happen by setting is-containment to false.


	NOTE: Both ownedItems and interestingItems are set to type TopologyObject. Remember that TopologyObject is the base type for most objects in Foglight. By using this as the type for a collection, we are allowing nearly any object to be added to the collection. In this case, it makes sense, as it allows for an Employee to include services, hosts, application servers, databases, and agents in their ownedItems list.

There is one last technique to show related to model definitions. The value of a property can be set at run-time by agent or script, but we can also supply code to create a property. As a simple example, consider the fact that we have stored lastName and firstName separately. It is likely that we want to put the two names together when we build dashboards. To do that, we could add a fullName property that has a copy of the data in lastName and firstName, but they might get out of sync. It makes much more sense for us to make fullName a derived property, and to provide a Groovy code to calculate its value:

<value>

StringBuilder fullName = new StringBuilder();

fullName.append(scope.get("firstName")).append(" ");

fullName.append(scope.get("lastName"));

return fullName.toString();

</value>

</annotation>

</property>

The code is straight forward and should be familiar to everyone who knows either Groovy or Java®. We are building a string that puts the first and last names together, and returns that value. When we access fullName, this script is run.


	NOTE: We need to emphasize that the value of an object’s Property can be produced by a Groovy expression. This is a powerful capability. For example, you can create a property that finds all SQL Server® database instances running on hosts that are also running WebLogic.

The “Manager” type

The Manager type extends Employee; its simplified definition contains one property, directReports.

</type>

This adds the ability to have a list of direct reports. By setting the is-many attribute to true, we allow a Manager to have zero or more Employees.

The “Organization” type

The Organization type (simplified) definition contains several properties:

</type>

An Organization inherits TopologyObject. Each Organization has an orgName that uniquely defines its identity. All Organization instances must have different values for orgName.

Each Organization has a relationship with a Manager through the orgLeader property. Each Organization can only have one manager. This might be a flaw in our model, but we will keep it as-is, for now. Also note that is-containment is set to true. This means that an Organization inherits the state of its Manager, who in turn inherits the state of his or her Employees.

Each Organization also has a relationship with zero or more child Organization objects through the childOrgs property. This property has is-many attribute set to true, in order to allow an Organization to have zero or more child Organizations.

How models hang together

Objects can be created easily in Foglight. However, unless some thought is given to their structure, objects can be easily lost or become orphans.

To find orphan objects:

On the navigation panel, under Dashboards, click Configuration > Data.

The Data Browser dashboard appears.

Expand the topology tree to show the Management Server > All Data > All Type Instances > TopologyObject collection.

Figure 6. Expanded topology tree

Underneath this collection you can find the list of all instances organized by type.

Open TopologyObject > Sub Types, then (for example) open Organization to see all Organization object instances.

But this is the hard way to find things. And if your objects are orphaned, you cannot write dashboards. What you want to do is to anchor all your new objects into a collection somewhere. To do that, you need a type definition as follows:

</type>

This creates a model that contains all Organization instances. As we create our objects (see Full definitions for the org structure types), we need to make sure that we add them to OrganizationModel.organizations. But how does this help? The key is the ModelRoot. This is a special marker class that tells Foglight to include this model in the list of ModelRoots. This means it displays as shown in the following illustration.

Figure 7. OrganizationModel property in the topology tree