-
Title
How to force Foglight to use TCP when in High Availability (HA) mode? -
Description
After experiencing local UDP problems, how can one force Foglight to use TCP in High Availability (HA) mode?
-
Resolution
TCP HA Configuration steps
This procedure is documented in the HA Field Guide.
1. Shutdown the server
2. After FMS 5.9.1(Foglight Management Server) edit on each FMS the following file: $FMS_HOME/config/jgroups-config.xmlNOTE: As of Foglight 5.7.1 through 5.7.5, modify the file located in %FGLHOME%/server/jboss/deploy/cluster-service.xml
3. Follow procedure in the guide to Reconfigure JGroup to use TCP instead of UDP.
4. Start the primary server
5. Wait for the primary server to start up and then restart the secondary server(s).
6. Please see attached samples for both servers, primary and secondary.
Note: Examples attached is how you can comment/uncomment the fields properly.
For this exercise example, it was defined the following IP Addresses for HA:
Primary: 192.168.0.1Secondary: 192.168.0.2
The following example picture in the left side has all the changes applied, everything that is in blue is what was changed or added. And in the right hand file which is the original file with no changes, everything that is in red, that is what needs to be removed
How to check that HA is properly configured
Having these variables defined:SERVER1(192.168.0.1)SERVER2(192.168.0.2)After shutdown Foglight services on SERVER1 and while reviewing the Foglight Management Server logs on the SERVER2 which becomes the primary server, you can see the following from support_bundle_management_server.zip and ManagementServer_001.log:
---------------------------------------------------------GMS: address is x.x.x.x:7800 (cluster=FMS_HA)---------------------------------------------------------2017-04-23 12:59:30.040 VERBOSE [forge-startup] org.jboss.ha.framework.interfaces.HAPartition.FMS_HA - Number of cluster members: 22017-04-23 12:59:30.040 VERBOSE [forge-startup] org.jboss.ha.framework.interfaces.HAPartition.FMS_HA - Other members: 1..2017-04-24 00:06:41.320 VERBOSE [VERIFY_SUSPECT.TimerThread] org.jboss.ha.framework.interfaces.HAPartition.FMS_HA - Suspected member: 192.168.0.1:78002017-04-24 00:06:41.445 VERBOSE [IncomingMessageHandler (channel=FMS_HA)] org.jboss.ha.framework.interfaces.HAPartition.lifecycle.FMS_HA - New cluster view for partition FMS_HA (id: 2, delta: -1) : [SERVER2:1099]2017-04-24 00:06:41.445 VERBOSE [AsynchViewChangeHandler Thread] org.jboss.ha.framework.server.DistributedReplicantManagerImpl.FMS_HA - I am (SERVER2:1099) received membershipChanged event:2017-04-24 00:06:41.445 VERBOSE [AsynchViewChangeHandler Thread] org.jboss.ha.framework.server.DistributedReplicantManagerImpl.FMS_HA - Dead members: 1 ([SERVER1:1099])2017-04-24 00:06:41.445 VERBOSE [AsynchViewChangeHandler Thread] org.jboss.ha.framework.server.DistributedReplicantManagerImpl.FMS_HA - New Members : 0 ([])2017-04-24 00:06:41.445 VERBOSE [AsynchViewChangeHandler Thread] org.jboss.ha.framework.server.DistributedReplicantManagerImpl.FMS_HA - All Members : 1 ([SERVER2:1099])2017-04-24 00:06:41.445 INFO [AsynchViewChangeHandler Thread] com.quest.nitro.service.ha.HAMembershipService - Running as the primary server of the partition FMS_HA with 1 servers in total.Note: Attached to this KB you can see two attachments related to Primary and Secondary configuration examples and one more attachment with no modifications at all with the Out of the Box settings.
IMPORTANT NOTE: If using Foglight HA with SSL(https) and after configured HA mode, but not working, please make sure https is not configured on restart_monitor.config
In order to do that check this solution: "The FMS in HA mode keeps restarting" SOL68688
NETWORK TROUBLESHOOTING STEPS
1). Do nslookup of the SERVER1 on SERVER2 and viceverse to make sure DNS resolution is working (If needed use fully qualified domain name):
nslookup SERVER1
nslookup SERVER2
2). These are the ports that need to be opened on both server bi-directional, make sure that the ports are listening:
45566
7800
On Windows on both servers run
netstat -an
On Linux on both servers run
netstat -tlnp
Note: when running the above command on Linux, please do it using root user and provide the output, otherwise you will get the following message:
(Not all processes could be identified, non-owned process info will not be shown, you would have to be root to see it all.) -
Attachments
-
Additional Information
How to configure Remote Monitor to report on the availability of the Foglight Management Server? SOL99173