Foglight 5.9.1 - Performance Tuning Field Guide

High Availability (HA) Tuning

This chapter provides information about the high availability related options that can have an effect on performance, and describes the applicable performance-related options.

Tuning Connection Issues in an HA Implementation

High Availability (HA) means running multiple Foglight™ Management Servers in a cluster, where one is the primary and one or more are secondaries (on standby). The general setup, as well as the communication between the members of the cluster, requires special attention.

In some cases, the communication between servers in the cluster is not reliable. Sometimes unwanted behaviors occur, such as:

These issues may be attributable to JGroup (the underlying communication package that JBoss® uses for its HA implementation) and to the fact that a Foglight HA implementation uses UDP for communication by default and UDP is by nature an unreliable protocol.

If you encounter these HA issues, consider performing the following tuning exercise.

NOTE: If the system clock is out of sync by 5 seconds or more during HA server startup, a warning message similar to the following appears in the log: Warning: system time not in sync with primary server, Mon Aug 30 11:47:57 ECT 2010 vs Mon Sug 30 11:48:52 EDT 2010.)

NOTE: You must make the following changes to each server in the cluster.

Managing Hosts with Multiple Network Interfaces

When configuring HA servers that are installed on hosts with multiple network interfaces (that is, with multiple IP addresses and host names), you must specify the IP address that is to be used for communication between the servers.

To specify the IP address or host name to be used for communication, set the following:

--host=<your_ip_or_hostname>

You can set this using the command line interface or by adding the following to the server.config file:

server.cmdline.option0 = "-Djboss.bind.address=127.0.0.1";

or

server.cmdline.option0 = “-Dbind.address”;

server.cmdline.option1 = “127.0.0.1”;

where 127.0.0.1 is the desired IP address

or

server.cmdline.option0 = "-Djboss.bind.address=host1.example.com";

server.cmdline.option1 = "-Dbind.address=host1.example.com";

where host1.example.com is the desired host name

You also need to reconfigure the restarter to perform server health check with the same host name or IP address. To do so, open the restart_monitor.config file, find the line beginning with health.check.url, and replace localhost with the host name or IP address specified with --host.

In addition, an HA configuration has a feature that automatically redirects HTTP requests from secondary servers to the primary server. If the primary server has multiple network interfaces, you must specify the host name that is to be used to serve HTTP requests. You can do this using the command line interface, or by adding the following to the server.config file:

-Dquest.host.name=”host1.example.com”

JDK with IPv6 on Linux

Foglight™ Management Servers running on a VMware® Linux® x86_64 image fail to start up and a "java.net.BindException: Cannot assign requested address exception" is raised.

This is a JDK issue with IPv6 on Linux, not a true High Availability issue. It also impacts standalone servers. However, the issue is documented here because it occurs when the JBoss® cluster service is starting up.

Workaround: Start the Management Server with the following command:

-Djava.net.preferIPv4Stack=true

For more information, see: the JGroups FAQ (http://community.jboss.org/wiki/JGroupsFAQ) and Oracle® Bugs for it (http://bugs.sun.com/bugdatabase/view_bug.do?bug_id=6521974).