-
Title
Unable to Migrate Database agents due to host performance or high resource usage of RAM or CPU. -
Description
Unable to Migrate Database agents due to host performance or high resource usage of RAM or CPU. Migration fails to validate or complete the move selected Database or Database and Infrastructure agents if selected.
- Error in ManagementServer_xxx.log:
INFO [cvxpkghgbz:866449332143284270042863] com.quest.qsi.fason.framework.installer.impl.InstallerAgentServiceImpl - Creating HOST\INSTANCE agent of type DB_SQL_Server on FglAM host2.domain.com.VERBOSE [cvxpkghgbz:866449332143284270042863] com.quest.glue.adapter.service.CreateAgentHandler - Sending create message for agent new agent instance 1b6141ed-5096-369b-a2a7-8f6eebf23fe9 on host host.domain.comERROR [FglAM:IncomingMessageResponse[4]-237] com.quest.glue.adapter.service.CreateAgentHandler - The creation of agent 1b6141ed-5096-369b-a2a7-8f6eebf23fe9 on host.domain.com was not successful. The agent was not created. Cause: java.lang.IllegalArgumentException: The AgentID 1b6141ed-5096-369b-a2a7-8f6eebf23fe9 is already assigned to another agent instanceERROR [cvxpkghgbz:866449332143284270042863] script.system:dbwc_mssql_installer_migration.migrateAgents - Agent named HOST-INSTANCE with ID 13ERROR [cvxpkghgbz:866449332143284270042863] script.system:dbwc_mssql_installer_migration.migrateAgents - Failed to migrate agent HOST-INSTANCEcom.quest.qsi.fason.framework.installer.exceptions.InstallerServiceException: Could not create agent DB_SQL_Server on FglAM host.domain.com: Failed to create an agent DB_SQL_Server null- On the FMS or FglAM host high memory usage (90%+), especially if the embedded FglAM is at or near max recommended agents.
- Other processes present on the FMS or FglAM competeing for signifigant resources(SQL Server, 3rd party tools, etc)
- The JVM heap is set lower than recommended for Foglight componenets. [ FMS: server.config, FglAM (older versions) vm.config, (newer versions) baseline.jvmargs.config ]
-
Cause
CAUSE 1
Low free RAM
example: 98% constant usage which is inadequate when bulk operations trigger or during peak business hours.CAUSE 2
Java Heap (-Xmx, -Xms) exhaustion
example: 150 agents on Agent Manager with 1024M heap set ( -Xms/-Xmx parameters )CAUSE 3
Memory reservations are not set on the Virtual Machine
-
Resolution
Please refer to the Foglight for Databases Deployment Guide for current architecture sizing information.
RESOLUTION 1
Install more Physical RAM in the host, or if host is a VM, allocated additional RAM to the VM.
RESOLUTION 2
Modify the JVM heap on the Foglight Agent Manager as per the Foglight for Databases deployment guide recommendations
To change the memory settings:
- Connect to the host with the Agent Manger which requires update
- Open and Edit the Agent Manager JVM settings in the baseline.jvmargs.config file located under \state\default\config
- Update or add the following entries based on the recommended sizing(example below):
vmparameter.0 = "-Xms2048m";
vmparameter.1 = "-Xmx2048m";
Note 1: Ensure the host has enough RAM, as this increase may compete for resources with other processes on the host.
Note 2: The file may need to opened as a user who has permissions in order to save it(ex. Admin/Owner rights). - Restart the Foglight Agent Manager
An additional tuning step is: Increase the disk space parameter of the FglAM, following the steps below:
- Go to $FglAM_HOME/config/ and make a backup copy of file fglam.config.xml.
- Open the file fglam.config.xml and search for the parameter "max-disk-space"
- Change the values like listed below:config:upstream attribute max-disk-space="102400"
config:downstream attribute max-disk-space="102400" - Save the file and restart the FglAM
RESOLUTION 3
Set virtual machine reservations as described in KB 176968
A restart of either the FMS, FglAM or a reboot of the host is usually required to reinitialize the Foglight processes to use the new memory.