Agent Package Deployment Failure Due to JVM Heap Allocation Limits (4341575)

Issue Summary

After restarting the Foglight Agent Manager (FglAM), some agents fail to activate. Affected agent packages show deployment errors, indicated by:

• A directory under [FGLAM_HOME]/agents/[AGENT_PACKAGE_NAME] with a suffix like _ERROR_[TIMESTAMP]
• A agent.manifest file renamed to agent.manifest_error

Example:
The HostAgents directory contains HostAgents_ERROR_1684251567135, and inside it, agent.manifest_error. The same issue is observed with the DB_Oracle agent package.

The issue is caused by insufficient physical memory to accommodate the JVM heap size increase during agent package deployment.

Error Log Snippet:

ERROR [Startup Thread] DeploymentManager - Failed to build an AgentPackage...
Cause: The addition of 16 MB to the negotiated JVM Max heap size would adjust to 28 GB, which exceeds the total available physical memory of 26 GB. Rejecting memory request.

Note: Correcting the heap size in baseline.jvmargs.config alone is not sufficient if the error state has already occurred. Other undocumented causes may also lead to this issue. 

1. Stop the FglAM daemon/service
   [FGLAM_HOME]/bin/fglam -q
2. Update JVM heap settings
   Edit [FGLAM_HOME]/state/default/config/baseline.jvmargs.config and ensure heap size values are within physical memory limits.
3. Clean up deployment state
   Delete the directory:
   [FGLAM_HOME]/state/default/config/deployments
   This will be automatically recreated on FglAM restart.
4. Rename error directories and files
   Navigate to [FGLAM_HOME]/agents/[AGENT_PACKAGE_NAME] and:
   • Rename the [VERSION]_ERROR_[TIMESTAMP] directory to remove the error suffix
   • Inside [VERSION]/config, rename agent.manifest_error to agent.manifest
5. Repeat Step 4 for each affected agent package
6. Restart the FglAM daemon/service
   [FGLAM_HOME]/bin/fglam -d

• These steps apply to Linux-based FglAM installations. For Windows installations restart the Foglight service
• Always validate memory settings against available system resources before restarting FglAM.