The FMS (Foglight Management Server) in HA (High Availability) mode keeps restarting.
In the FMS logs a shutdown request can be seen before the event; for example:
VERBOSE [Thread-168] main\src\native\Launcher\src\windows\control - Shutdown request from: SYSTEM. Process ID 6556
Messages of the following type may be found in the server restarter log (server_restarter_*.log):
WARN [server-heartbeat-listener] com.quest.nitro.ha.monitor.local.ServerHeartBeatListener - Forge Server has not responded to health checking 761 seconds after startup. Next retry in 60 seconds. INFO [Thread-2] QcnUtil\src\windows\qcn_service - Received stop request. the FoglightHA service will now be shut down. INFO [Thread-4] QcnUtil\src\windows\qcn_process_windows - Shutdown request transmitted.
ERROR [server-heartbeat-listener] com.quest.nitro.ha.monitor.local.ServerHeartBeatListener - Forge Server is not yet available 5,401 seconds after startup, grace time is up and the server will be restarted. INFO [server-heartbeat-listener] com.quest.nitro.ha.monitor.local.ServerHeartBeatListener - Stopping Forge Server. INFO [server-heartbeat-listener] QcnUtil\src\windows\qcn_process_windows - Shutdown request transmitted. INFO [server-heartbeat-listener] com.quest.nitro.ha.monitor.local.ServerHeartBeatListener - Forge Server stopped normally. INFO [server-heartbeat-listener] com.quest.nitro.ha.monitor.local.ServerHeartBeatListener - Attempting (1/1) to restarting Forge Server with the command bin\fms -Dfoglight.cluster.mode=true -Dquest.common.process-runner=false -Dquest.native.launcher.io=true ... INFO [server-heartbeat-listener] com.quest.nitro.ha.monitor.local.ServerHeartBeatListener - Forge Server reports normal health state.
VERBOSE [server-heartbeat-listener] com.quest.nitro.ha.monitor.local.ServerHeartBeatListener - Fail to check Forge Server health state. org.springframework.remoting.RemoteConnectFailureException: Could not connect to HTTP invoker remote service at [http://localhost:8080/foglight-sl/HealthCheck]; nested exception is java.net.ConnectException: Connection refused: connect ... Caused by: java.net.ConnectException: Connection refused: connect
VERBOSE [server-heartbeat-listener] com.quest.nitro.ha.monitor.local.ServerHeartBeatListener - Fail to check Forge Server health state. org.springframework.remoting.RemoteAccessException: Could not access HTTP invoker remote service at [https://localhost:8443/foglight-sl/HealthCheck]; nested exception is javax.net.ssl.SSLHandshakeException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
CAUSE 1
The health check URL is configured to use HTTPS and the certification path cannot be validated.
CAUSE 2
The FMS isn't listening on the localhost address. The FMS host has multiple IP addresses and bound to one of them using the steps from KB article 4246515.
CAUSE 3
The default HTTP port was modified.
CAUSE 4
The restart monitor gives the server a grace period of 60 minutes by default; if the FMS takes longer to start up the health check fails.
CAUSE 5
The upgrade process was taking a very long time enabling the cartridges and during this time, the HA server restarter has already exceeded the grace period to which it has to wait for the FMS startup.
RESOLUTION 1
Change the restart monitoring configuration to use HTTP:
Check the [fms_home]/config/restart_monitor.config file to see if the health check URL is set as follows:
health.check.url = "https://localhost:8443/foglight-sl/HealthCheck";
If the health check URL is using https modify the value of the health.check.url parameter and set it as follows:
health.check.url = "http://localhost:8080/foglight-sl/HealthCheck";
After modifying the health.check.url parameter the change should be reflected in the server restarted logs; if not, stop the FMS, verify that the FMS is stopped, and start the FMS in HA mode.
Another option is to import the certificates to the JRE TrustStore and update the health check URL on both nodes with their respective node/host name; for example:
[fms_home]/jre/bin/keytool -import -trustcacerts -alias [alias_of_cert] -keystore [fms_home]/jre/lib/security/cacerts -storepass changeit -file [path_to_cert_file]
In restart_monitor.config:
health.check.url = "https://foglight.yourdomain.com:8443/foglight-sl/HealthCheck";
Note: The JRE on disk is replaced completely during a Management Server upgrade. Any existing certificates are retained in [fms_home]/state/backup/[previous-fms-version]/jre/lib/security/cacerts.
RESOLUTION 2
Modify the health.check.url in the [fms_home]/config/restart_monitor.config file to use the bound IP address as follows:
health.check.url = "http://boundip:8080/foglight-sl/HealthCheck";
RESOLUTION 3
Within the [fms_home]/config/restart_monitor.config file change the health.check.url value to match the non-default HTTP port specified in [fms_home]/config/server.config.
Example:
In server.config:
server.http.port = "8181";
In restart_monitor.config:
health.check.url = "http://hostname:8181/foglight-sl/HealthCheck";
RESOLUTION 4
If the FMS takes longer than the default grace period, consider increasing the start.grace parameter in the [fms_home]/config/restart_monitor.config file to a value greater than the amount of time it takes the FMS.
Default value is:
startup.grace = 3600;
RESOLUTION 5
Complete the following steps to increase the startup grace period:
startup.grace from 3600 to 7200 for the file [fms_home]/config/restart_monitor.config:
startup.grace = 7200;