-
Title
SQL Server database cartridge does not enable during upgrade -
Description
After upgrading the SQL Server cartridge, the SQL Server cartridge does not enable. This can result in being unable to view SQL Server Databases within the Databases dashboard.
Error messages similar to the following are populating within the Foglight Management Server (FMS) log file:
INFO [forge-startup] com.quest.nitro.service.cartridge.api.CartridgeManager - Successfully re-enabled cartridge: DB_Global_View_UI-5_7_5_50 on startup.
WARN [CARTRIDGE_OPERATION-0] com.quest.nitro.service.cartridge.api.CartridgeManager - Cannot enable cartridge DB_SQL_Server_UI-5.7.5.50 on because of missing dependencies com.quest.nitro.service.cartridge.api.CartridgeDependencyException: Missing dependency for Cartridge: DB_SQL_Server_UI-5.7.5.50 - 5.7.5.50-20180321-1057:DB_SQL_Server-5.7.5.50 (matches: LATEST).
com.quest.nitro.service.cartridge.api.CartridgeDependencyException: Missing dependency for Cartridge: DB_SQL_Server_UI-5.7.5.50 - 5.7.5.50-20180321-1057:DB_SQL_Server-5.7.5.50 (matches: LATEST)WARN [Transaction Reaper Worker 0] com.arjuna.ats.arjuna - ARJUNA012381: Action id 0:ffff0ae0a255:d2d6:5b114395:82b2e completed with multiple threads - thread CARTRIDGE_OPERATION-0 was in progress with org.hibernate.impl.SessionImpl.getEntityMode(SessionImpl.java:1309)
ERROR [CARTRIDGE_OPERATION-0] script.system:administration_cartridges_cartridgeinventory.enableCartridgeTask - com.quest.nitro.service.cartridge.api.CartridgeDependencyException: Missing dependency for Cartridge: DB_SQL_Server_UI-5.7.5.50 - 5.7.5.50-20180321-1057:DB_SQL_Server-5.7.5.50 (matches: LATEST)
WARN [Transaction Reaper Worker 0] com.arjuna.ats.arjuna - ARJUNA012381: Action id 0:ffff0ae0a255:d2d6:5b114395:82b2e completed with multiple threads - thread CARTRIDGE_OPERATION-0 was in progress with org.hibernate.impl.SessionImpl.getEntityMode(SessionImpl.java:1309)
WARN [Transaction Reaper Worker 0] com.arjuna.ats.arjuna - ARJUNA012108: CheckedAction::check - atomic action 0:ffff0ae0a255:d2d6:5b114395:82b2e aborting with 1 threads active!
WARN [CARTRIDGE_OPERATION-0] com.arjuna.ats.arjuna - ARJUNA012077: Abort called on already aborted atomic action 0:ffff0ae0a255:d2d6:5b114395:82b2e
WARN [Transaction Reaper] com.arjuna.ats.arjuna - ARJUNA012117: TransactionReaper::check timeout for TX 0:ffff0ae0a255:d2d6:5b114395:82b2e in state CANCEL
WARN [Transaction Reaper] com.arjuna.ats.arjuna - ARJUNA012378: ReaperElement appears to be wedged:
com.arjuna.ats.arjuna.coordinator.TransactionReaper.doCancellations(TransactionReaper.java:403)
com.arjuna.ats.internal.arjuna.coordinator.ReaperWorkerThread.run(ReaperWorkerThread.java:78)ERROR [CARTRIDGE_OPERATION-0] com.quest.nitro.service.cartridge.api.CartridgeManager - Cartridge DB_SQL_Server-5.7.5.50 could not be enabled. Enabling component DB_SQL_Server-Properties-5.7.5.50 failed with error: java.lang.IllegalStateException: java.lang.IllegalStateException: Transaction is not active: TransactionImple < ac, BasicAction: 0:ffff0ae0a255:d2d6:5b114395:82b2e status: ActionStatus.ABORTING >, status: 9. All components of DB_SQL_Server-5.7.5.50 will be disabled.java.lang.IllegalStateException: java.lang.IllegalStateException: Transaction is not active: TransactionImple < ac, BasicAction: 0:ffff0ae0a255:d2d6:5b114395:82b2e status: ActionStatus.ABORTING >, status: 9 -
Cause
High amount of processing is occurring from the following lifecycle scripts
VERBOSE [CARTRIDGE_OPERATION-0] script_ebe24b042c6b78b29079ba97389cebca - DB SQL-Server Topology Upgrade - INFO:Upgrade Topology life cycle called (cartridgeName=DB_SQL_Server, cartridgeVersion=5.7.5.50)
VERBOSE [CARTRIDGE_OPERATION-0] script_76016a6ad72703b08058dedcaa995e88 - Running script activate Global_View cartridge
VERBOSE [CARTRIDGE_OPERATION-0] script_76016a6ad72703b08058dedcaa995e88 - sql-server-life-cycle-script(activate_gv_cartridge.groovy) Execution Duration [SEC]:0There are a large number SQL Server agents so that system may have frequent changes on the Agent's ASP (Agent Status Properties) that cause the persistable tables to be large. When upgrading the cartridge, the ASP would be updated also.
1. Many ASP values involved which are affected by the number of agents2. This may be slow to query/update on the persistable tables because of its size, in turn it may require a larger JDBC transaction time out value.3. This may take up more JDBC connections due to the slowness on querying/updating persistable tables(See #2) -
Resolution
In the FMS version 5.9.3 the default Transaction Coordinator timeout was increased as part of change request FGL-19818 from 10 minutes (600 seconds) to 20 minutes (1200 seconds).
This change may however be insufficient for some FMS systems where there are a large number of SQL Server agents. The Transaction Coordinator timeout may have to be temporarily increased even higher to allow for the upgrade to complete.
1. Restart the FMS backend database
a. Stop the Foglight Service
b. Restart the backend database
c. Do not start the Foglight service until the next step is completed2. Increase the number of connections to the FMS repository database as described in KB 93812
3. Review the number of ASP configuration objects in the Diagnostic Snapshot (found in a FMS support bundle) as described in KB 329765
4. Apply the following configuration parameter in the Foglight Management Server "{installation location}\Foglight\config\server.config" configuration file to increase the timeout to 80 minutes (4800 seconds):
-Dcom.arjuna.ats.arjuna.coordinator.defaultTimeout=4800
For example: server.vm.option0 = "-Dcom.arjuna.ats.arjuna.coordinator.defaultTimeout=4800";
5. Save the changes
6. Start the Foglight Service
7. Attempt to enable cartridges "DB_SQL_Server" and "DB_SQL_Server_UI" (this can take some time)
If successful, Foglight will inform the user that this task is successful. Confirm if the issue is resolved by navigating to the Databases dashboard (please note: it may take some time for all DBs to populate), then
-
Additional Information
During the upgrade the FMS log file may be spammed with many error messages similar to the following because the SQL Server cartridge has not been enabled. Users should be able to deactivate the SQL Server agents or shutdown the FglAMs reporting to the FMS to prevent the log files from filling up: ERROR [FglAM:IncomingMessage[3]-14] com.quest.glue.adapter.service.handlers.TopologyDataHandler - No CDT could be found for the agent type DB_SQL_Server (DB_SQL_Server/VERSION/DB_SQL_Server/AGENTNAME on [HOSTNAME/IPADDRESS]), the submitted data sample will be discarded. Perhaps the cartridge containing the CDT for this agent was disabled or uninstalled?