Foglight Management Server (FMS) does not complete startup.
The FMS log seems to stall at "com.quest.nitro.service.topology.TopologyService - Topology Objects Loaded"
Attempted start the FMS with a switch to bypass "refresh_on_startup".
$FGLHOME/bin/fms -d -Dfoglight.persistence.management.skip_refresh_on_startup=true
This allowed the FMS startup to go further but then it halted at: "all derivation definitions are loaded"
Foglight datase is Oracle
Oracle 10.2 or Oracle 10g BugID:8544896
In Oracle 10.2.0.4, bugs such as 7385253 or 8544896 may be the cause of reuse object (RO) enqueue waits. Therefore, ensure that a current kcb merge fix is implemented.
In 10g, the wait event is called "enq: RO - fast object reuse".
Foglight database is an external MySQL. The backup is running the same time the Foglight maintenance job runs. So the backup process blocks tables and the Foglight server stucks on aquiring locks on tables for the maintenance run.
This was an Oracle problem and not related to Foglight.
The DBA performed a manual truncation of the table indicated in the Foglight log which was attempting to truncate obs_metric_aggregate_0074 and determined it was locked by Oracle's (RO) enqueue wait. Resolved this per Oracle's knowledge base for BugID:8544896.
Foglight started properly after this action.
Configure the backup and the maintenance job to run in different time frames. To get the Foglight server out of the stuck state without restarting it and/or the FMS DB, just kill the backup process.