Defect FAM-7175 is the cause of the problem.
FAM-7175: Check why some FglAMs have the hops value of 0 in server
The RemoteHosts.getAllHosts() method returns a list of Hosts. In some large and/or busy environments the output sometimes includes entries with "hops" values of 0. The hops value being 0 for some FglAMs prevents the FglAMs with the hops value of 1 from being upgraded.
This problem can occur for FglAM versions earlier than 5.9.1 in large busy environments (e.g. hundreds of fglams running and submitting).
Run the attached script (RetrieveUpgradeStatus.groovy) from the script console dashboard with "FglAM-Adapter" cartridge selected in the "Choose an Associated Cartridge" drop down list to determine if FAM-7175 is the cause of the problem. There will be some "OOPHops=0" entries in the output if the issue is indeed caused by FAM-7175.
1). Navigate to "Administration | Tooling | Script Console".
2). Select the Scripts tab, click the "Add" button.
3). Select "FglAM-Adapter" from the "Choose an Associated Cartridge" drop down list.
4). Paste the contents of the RetrieveUpgradeStatus.groovyscript in the "Enter Script Text" field.
5). Click the "Run" button at the bottom of the "Run Script" window.
6). Copy and paste the output to a text file.
7). Check to see if there are any "OOPHops=0" entries in the output. If there are "OOPHops=0" entries in the output then FAM-7175 is the cause of the problem.
WORKAROUND
The following workaround is available but upgrading to the V5.9.1 (or higher) FglAM cartridge is much simpler.
1. Raise the "max connection permits" value as described in KB article 152661.
2. Restart the FMS.
3. If the problem persists, execute steps #4, #5 and #6 below.
4. Stop all the FglAMs and restart the FMS
5. Start some of the FglAMs that cannot be upgraded (20 for example). The FglAMs that were started should show as upgradable in the Agent Managers page.
6. Upgrade the FglAMs, and then stop them again.
7. Repeat steps #5 and #6 for the rest of the FglAMs that cannot be upgraded.
STATUS
This issue was logged as Defect FAM-7175. The fix is included with the V5.9.1 Foglight Agent Manager cartridge.
