Topology churn fixes (4345216)

It is generally assumed that topology objects change very little over time. Observations are expected to be highly volatile over time.

The decision of whether to add a particular piece of data to the topology model or to treat it as an observation is made during cartridge development. This decision is expressed in the software using CDT configuration.

The server is generally optimized to handle stable topology models where topology changes are infrequent. If topology changes occur on a more regular basis, this is known as Topology Churn, and it usually results in diminished server performance.

For more information refer to section "Topology Changes and Topology Churn" in the Foglight  - Performance Tuning Field Guide.

Identifying topology churn

In order to identify Topology Churn observed on Performance Report as High Topology changes, please check the following information:

1. How to identify a topology churn in the Diagnostic Snapshot file? (KB 4353149)

2. Foglight script to display the objects that are causing topology object "churn" (KB 4314685)

   When you open script "find-churning-objects_KB188003", you will see the following variables:

   typeName = "Host"; (the topology object to report on)
   numDays = 1; (the time period the script queries)
   churnThreshold = 10; (the threshold of the number of changes to an object to report on. Setting this to 0 will report everything)
   numObjects = 10; (the list of entries reported in the output)

   Note: "typeName" can be changed based on the most volatile objects seeing from Diagnostic Snapshot, these are common types that could be churning:

   For Infrastructure:
   Host
   HostProcess

   For VMware:
   VMWVirtualMachine
   VMWSnapshot
   VMWSnapshotFolder
   PhysicalDisk
   VFolder
   VMWVirtualMachineHost
   HostStorage
   VMWVirtualMachineLogicalDisk
   VMWVirtualMachineHostCPUs
   HostCPUs
   Processor
   LogicalDisk
   VMWResourcePool

3. The attached script "find-frequent-changes_4hours_hostprocess.groovy" is set to check for frequent topology changes during the last 4 hours; the "typeName" variable needs to be changed to the desired topology type.

4. After changes have been implemented to clean up any topology types causing high topology churn, the attached script "find-frequent-changes_15min_hostprocess.groovy" can run to find frequent changes for those affected types, but this time for the last 15 minutes to see the behavior for shorter time. The script was updated with startTime = now - (15 * 60 * 1000L);

Reviewing duplicate monitoring and churn for topology types

After identifying topology churn, different solutions may apply depending on the impacted agents and topology objects; please review the following articles:

For duplicate Infrastructure agents:

Duplicate Host displayed under Infrastructure Dashboard (KB 4305727)

Also check for Host aliases: 

No data displayed and no Host object created for host monitored by Infrastructure agent that appears to be collecting data (KB 4261921)

For FileLogMonitorFileGroup objects:

Churn in FileLogMonitorFileGroup topology type (KB 4253865)

For duplicate IP addresses:

Topology churn caused by IP addresses (KB 4275948)

For database agents and objects:

High churn and slow performance in FMS from MongoDB agents (KB 4368617 )

For VMware agents and objects: 

How to identify Virtual Machines with duplicate vmID (KB 4292241)

How to identify Virtual Machines with duplicate UUID (KB 4280308)

Cloned virtual machine with same UUID can cause a Foglight topology churn (KB 4257931)

Churn on VMWResourcePool (KB 4292556)

For MultiHostProcessMonitorAgent:

Multihost Agent host processes are missing (KB 4215085)