Return
-
Title
Foglight Agent Manager stops working with "message queue is full" -
Description
Foglight Agent Manager (FglAM) stops working. After restarting the FglAM, it works for sometime and then stop working again.
1. In Agent Manager log file, the following error appears intermittently:
WARN [CachedOutgoingQueueWorker] com.quest.glue.common.comms.CachedOutgoingQueue - Removed 68 messages because the message queue is full.
2. From Foglight UI, under Administration │ Agents │ Agent ManagersSee if there is one alarm with this message:
"The agent manager has removed x messages because the Upstream-data cache is full. The removed data messages cause gaps in data. To fix this, please increase the upstream max-disk-space setting for this agent manager. -
Cause
Max disk space is set to low value and need to be increased. -
Resolution
WORKAROUND
1) Stop FglAM service/daemon.2) Delete the contents inside $FglAm_Home/state/default/cache
3) Change the values shown below to increase the message cache size for both upstream and downstream from 1 MB to about 10240 MB, and upstream-verified from 512 KB to 4096 MB.The upstream/downstream max-disk-space values are tuned to impact the system as little as possible in most use cases.
Edit the file under $FglAm_Home
/state/default/config/fglam.config.xml and make the following changes and save the file. Existing:
<config:upstream max-queue-size="-1" max-disk-space="1024" max-batch-size="500" allow-runtime-change="true"/><config:upstream-verified max-queue-size="-1" max-disk-space="512" max-batch-size="250" allow-runtime-change="true"/><config:downstream max-queue-size="-1" max-disk-space="1024" max-batch-size="500" allow-runtime-change="true"/>
Change To:
<config:upstream max-queue-size="-1" max-disk-space="10240" max-batch-size="500" allow-runtime-change="true"/><config:upstream-verified max-queue-size="-1" max-disk-space="4096" max-batch-size="250" allow-runtime-change="true"/><config:downstream max-queue-size="-1" max-disk-space="10240" max-batch-size="500" allow-runtime-change="true"/>
Confirm that the change were applied properly.4) Delete the existing deployed negotiation configuration settings inside directory:$FglAm_Home/state/default/config/deployments 5) Restart the Agent Manager for these settings to take effect.