"Message queue is full" causes Foglight Agent Manager (FglAM) performance issues (4308485)

This article explains how to improve Foglight Agent Manager (FglAM) performance when experiencing disconnections from the Foglight Management Server (FMS), system slowness, or "message queue is full" warnings in the FglAM log file.

In some cases, the FglAM may stop working after running for some time. Restarting the FglAM temporarily resolves the issue, but it may recur.

Symptoms

• Intermittent error in Agent Manager log file:

  WARN [CachedOutgoingQueueWorker] com.quest.glue.common.comms.CachedOutgoingQueue - Removed 68 messages because the message queue is full. 

• Foglight UI Alarm under Administration │ Agents │ Agent Managers:

  The agent manager has removed x messages because the Upstream-data cache is full. The removed data messages cause gaps in data. To fix this, please increase the upstream max-disk-space setting for this agent manager.

The configured max disk space for message queues is too low and needs to be increased.

Workaround steps:

1. Stop the FglAM service/daemon.

2. Delete all files inside the cache directory:

    

   $FglAM_Home/state/default/cache

3. Edit the configuration file:

    

   $FglAM_Home/state/default/config/fglam.config.xml

   Update the message cache sizes for upstream, upstream-verified, and downstream queues from their default values to larger ones as follows (all values in KB):

   Current settings:

   <config:upstream max-queue-size="-1" max-disk-space="1024" max-batch-size="500" allow-runtime-change="true"/> <config:upstream-verified max-queue-size="-1" max-disk-space="512" max-batch-size="250" allow-runtime-change="true"/> <config:downstream max-queue-size="-1" max-disk-space="1024" max-batch-size="500" allow-runtime-change="true"/>

   Change to:

    

   <config:upstream max-queue-size="-1" max-disk-space="102400" max-batch-size="500" allow-runtime-change="true"/> <config:upstream-verified max-queue-size="-1" max-disk-space="40960" max-batch-size="250" allow-runtime-change="true"/> <config:downstream max-queue-size="-1" max-disk-space="102400" max-batch-size="500" allow-runtime-change="true"/>

   Save and verify changes.

4. Delete all contents in the deployments directory:

    

   $FglAM_Home/state/default/config/deployments

5. Restart the FglAM service/daemon to apply changes.

Important Notes:

• Increasing these values will increase memory and disk space usage on the FglAM host.

• If "message queue is full" messages persist after these changes, repeat steps 1-5 and increase the max-disk-space values incrementally (e.g., add 20,000 to 50,000 KB at a time) until the issue resolves.