Understanding and Managing Topology Churn in Foglight Management Server (FMS) (4345216)

This article explains the impact of topology churn on Foglight Management Server (FMS) performance and stability. It outlines common symptoms, causes, and recommended actions for reducing or managing topology churn. 

• Slow or unresponsive Foglight dashboards

• Delayed or missed alarms and email notifications

• High FMS memory usage or frequent Full GC events

• Agent registration failures or missed data collections

• Support bundle generation failures or timeouts

• Duplicate or missing topology objects in the UI

Topology churn occurs when topology objects (e.g., database instances, hosts, VMs, network connections) are frequently created, updated, or deleted. This can be caused by:

• Flapping agents (e.g., due to connectivity or misconfiguration)

• Overactive or repeated discovery jobs

• Automation scripts frequently altering monitored assets

• High-frequency custom data collections that create new objects dynamically

• Concurrent agent provisioning or restarts

Churn increases memory, CPU, database I/O, and internal event processing overhead on the FMS, often resulting in degraded performance.

Effects on FMS

Resolution

1. Identify Churn Sources

• Review agents with high object creation rates (e.g., VMware, DB agents).
• Please check the following information:
  • How to identify a topology churn in the Diagnostic Snapshot file? (KB 4353149)
  • Foglight script to display the objects that are causing topology object "churn" (KB 4314685)
    • When you open script "find-churning-objects_KB188003", you will see the following variables:
      typeName = "Host"; (the topology object to report on)
      numDays = 1; (the time period the script queries)
      churnThreshold = 10; (the threshold of the number of changes to an object to report on. Setting this to 0 will report everything)
      numObjects = 10; (the list of entries reported in the output)
      Note: "typeName" can be changed based on the most volatile objects seeing from Diagnostic Snapshot, these are common types that could be churning:
    • For Infrastructure:
      Host
      HostProcess
    • For VMware:
      VMWVirtualMachine
      VMWSnapshot
      VMWSnapshotFolder
      PhysicalDisk
      VFolder
      VMWVirtualMachineHost
      HostStorage
      VMWVirtualMachineLogicalDisk
      VMWVirtualMachineHostCPUs
      HostCPUs
      Processor
      LogicalDisk
      VMWResourcePool

The attached script "find-frequent-changes_4hours_hostprocess.groovy" is set to check for frequent topology changes during the last 4 hours; the "typeName" variable needs to be changed to the desired topology type.

After changes have been implemented to clean up any topology types causing high topology churn, the attached script "find-frequent-changes_15min_hostprocess.groovy" can run to find frequent changes for those affected types, but this time for the last 15 minutes to see the behavior for shorter time. The script was updated with startTime = now - (15 * 60 * 1000L);

Reviewing duplicate monitoring and churn for topology types

After identifying topology churn, different solutions may apply depending on the impacted agents and topology objects; please review the following articles:

For duplicate Infrastructure agents:

• Duplicate Host displayed under Infrastructure Dashboard (KB 4305727)

Also check for Host aliases: 

• No data displayed and no Host object created for host monitored by Infrastructure agent that appears to be collecting data (KB 4261921)

For FileLogMonitorFileGroup objects:

• Churn in FileLogMonitorFileGroup topology type (KB 4253865)

For duplicate IP addresses:

• Topology churn caused by IP addresses (KB 4275948)

For database agents and objects:

• High churn and slow performance in FMS from MongoDB agents (KB 4368617 )

For VMware agents and objects: 

How to identify Virtual Machines with duplicate vmID (KB 4292241)

• How to identify Virtual Machines with duplicate UUID (KB 4280308)
• Cloned virtual machine with same UUID can cause a Foglight topology churn (KB 4257931)
• Churn on VMWResourcePool (KB 4292556)

For MultiHostProcessMonitorAgent:

• Multihost Agent host processes are missing (KB 4215085)

2. Reduce Load

• Pause or remove flapping or unused agents.

• • Alternatively, add the following JVM parameter to the baseline.jvmargs.config file and restart the FglAM:

    vmparameter.0 = "-Dagent.collector.schedule.load.max.delay.millis=300000";

    This spreads agent activation over 5 minutes instead of the default 2 minutes, reducing collection overlap.

    Stagger agent start times to reduce startup spikes by groups (20–30 agents every ~17 seconds)

• Avoid or refactor scripts or custom cartridges that create dynamic topology objects.

3. Monitor System Health

• Track heap usage and garbage collection

• Monitor CPU, thread usage, and disk I/O.

• Review Thread Dumps for long-held locks or slow tasks.

4. Restart FMS or FglAM

• If symptoms persist, perform a controlled restart of the FMS (and FglAMs if needed) during off-peak hours.

5. Tune JVM Memory

• If FMS memory usage is consistently >85%, increase heap size in server.config or service startup script.