'Upgrade Required' flag still shows in database dashboard after database agents wizard completed (4309024)

ISSUE 1, 2, and 6

Some database agents still show an "Upgrade Required" message after the database cartridge has been upgraded and the agents have been upgraded to the new cartridge version.

ISSUE 3 and 4 (Federator)

The upgrade link continues to appear in the database dashboard on a Federator even though all of the agents have been upgraded on each of the children FMS systems.

ISSUE 5 (Federator)

Agents missing from the child FMS systems are present on the Federator with the "Upgrade Required" message.

CAUSE 1

This problem can be caused by Foglight Agent Manager(s) (FglAMs) not running or are not connected to the FMS.

CAUSE 2

Issue with the "isUpgradeRequired" flag.

CAUSE 3

Issue was caused on a Federator by a corrupt agent on the child FMS. When attempting to view an agent log, or recycle the agent it would only hang.

CAUSE 4

Different database cartridge versions among the Federator and/or the database child member systems.

CAUSE 5

Database agents were deleted on the child servers of a Federator architecture only using the Agent Status panel. The Agent Model Data for these agents remain on the child FMS systems.

CAUSE 6

Dashboard got corrupted for any reason

RESOLUTION 1:

Ensure all associated FglAMs are running/started and connected to FMS after upgrading the FMS. It might be necessary to restart them. When they reconnect, the "Upgrade" option for the DB agents should be displayed at the top of the Database dashboard.

RESOLUTION 2:

Run the following groovy script from Administration | Tooling | Script Console | Scripts Tab | Add | Enter script text

For Oracle

def dataService = server.DataService;
def canonicalFactory = server.CanonicalFactory;
def agents = #!DBO_Agent_Model#.topologyObjects;
for (agent in agents)
{ dataService.publish(canonicalFactory.createNode(agent, "isUpgradeRequired", "false", 60000L)); }
return true;

RESOLUTION 3: 

On a Federator, the Agent Manager was initially restarted to allow the database agent to shut down. Then after the Agent Manager was restarted, the broken agent was deleted as per the steps in Solution SOL83829

The agent was then recreated on the child FMS, and after the child FMS synched up with the Federator, the database agent in the dashboard no longer had the "upgrade required" indicator.

RESOLUTION 4:

The customer upgraded all of the children and the Federator to use the same database cartridge versions and the issue was resolved.

RESOLUTION 5:

Please refer to KB 114545

RESOLUTION 6:

If the Foglight Agent Manager (FglAM) is offline or unavailable when the Database Agent installer/upgrade wizard runs following a cartridge upgrade, the database agents on the FglAM will not be upgraded.

When the FglAM is back online, the Database dashboard should display the upgrade wizard. If it does not, use the Administration | Agents | Agent Managers screen to manually upgrade the database cartridge on the FglAM.

RESOLUTION 7:

Reset the cartridge DB_Global_View_UI

1. Go to "Administration | Cartridges | Cartridge Inventory"
2. Find the Db_Global_View_UI cartridge and select it by ticking the box in the first column
3. Once selected click "Reset" on top of the table 

Cartridge upgrades consist of two phases

1. The first phase is installing the cartridge in Cartridge Inventory. This activity transfers the cartridge through the browser to the FMS cartridge manages the UI and basic functionality of the product.
2. The second phase, deploys a subset of the cartridge (a .gar file) from the FMS to the Agent Managers. It could be done through a wizard (as in the SQL Server, Oracle, and DB2 cartridges), or manually through the Administration | Agent Status or Administration | Agent Managers pages. This component on the FglAM is used for the communication processes between the local/remote host and the collections that run again the host.