-
Title
"Message queue is full" causes Foglight Agent Manager (FglAM) performance issues -
Description
How to improve Foglight Agent Manager (FglAM) performance in cases of disconnections from the FMS, slowness, or reported "message queue is full" messages are printed in the FglAM log file.
In some circumstances the Foglight Agent Manager (FglAM) may also stop working. After restarting the FglAM, it works for sometime and then stop working again.
1. In Agent Manager log file, the following error appears intermittently:
WARN [CachedOutgoingQueueWorker] com.quest.glue.common.comms.CachedOutgoingQueue - Removed 68 messages because the message queue is full.
2. From Foglight UI, under Administration │ Agents │ Agent ManagersSee if there is one alarm with this message:"The agent manager has removed x messages because the Upstream-data cache is full. The removed data messages cause gaps in data. To fix this, please increase the upstream max-disk-space setting for this agent manager. -
Cause
Max disk space is set to low value and need to be increased. -
Resolution
WORKAROUND
1) Stop FglAM service/daemon.2) Delete the contents in$FglAM_Home/state/default/cache
3) Edit the file$FglAm_Home/state/default/config/fglam.config.xml.
Change the values shown below to increase the message cache size for both upstream and downstream from 1024 to about 102400, and upstream-verified from 512 to 4096 (all values are in KB - Kilobytes)The upstream/downstream max-disk-space values are tuned to impact the system as little as possible in most use cases.
Existing:
<config:upstream max-queue-size="-1" max-disk-space="1024" max-batch-size="500" allow-runtime-change="true"/><config:upstream-verified max-queue-size="-1" max-disk-space="512" max-batch-size="250" allow-runtime-change="true"/><config:downstream max-queue-size="-1" max-disk-space="1024" max-batch-size="500" allow-runtime-change="true"/>Change To:
<config:upstream max-queue-size="-1" max-disk-space="102400" max-batch-size="500" allow-runtime-change="true"/><config:upstream-verified max-queue-size="-1" max-disk-space="40960" max-batch-size="250" allow-runtime-change="true"/><config:downstream max-queue-size="-1" max-disk-space="102400" max-batch-size="500" allow-runtime-change="true"/>
Save the file and confirm that the changes were applied properly.4) Delete the contents in$FglAM_Home/state/default/config/deployments5) Restart the Agent Manager for these settings to take effect.NOTE: These changes will require additional memory use on the FglAM server host. If "message queue is full" messages continue to appear in the FglAM log after these changes, repeat steps 1-5 while incrementally increasing the disk space values moderately (for example by 20000 to 50000 KB at a time)