Foglight Agent Manager 5.8.5.7 - Foglight Agent Manager Guide

Configuring the Agent Manager to work in HA mode

High Availability (HA) mode is a configuration in which multiple Agent Managers work together in an HA Partition, where one Agent Manager is a primary host (HA Primary), and others are standby hosts (HA Peers). When configured, agent instances whose types are configured as HA Aware and belong to the same HA Partition are managed by the HA Primary host. If that Agent Manager stops responding or goes offline, the agent instances fail over to another Agent Manager.

Under a configured HA Partition, a common deployment set of agent types is kept in sync across all HA Peers. Agent packages deployed to that HA Partition are checked for any HA Aware agent types. Any detected HA Aware types are automatically deployed to all other HA Peers.

The FglAM Adapter monitors the deployments of the each Agent Manager host within the named HA Partition. The HA Primary is considered the master in terms of the deployment set and automatically deploys (or undeploys) HA Aware cartridges to each HA Peer. This also happens during cartridge upgrades, when the Adapter automatically pushes out the updates to all of the HA Peers in that HA Partition.

 

CAUTION: Do not create Agent Manager HA clusters on Agent Manager installations that already contain agent packages. Doing so causes any HA-aware packages, that only exist on the secondary node, to become automatically undeployed. This is because the primary node controls the deployment all HA-aware packages.

Assigning Agent Managers to HA partitions

HA mode is configured through the FglAM Adapter agent properties. You can use these properties to assign an Agent Manager to HA partitions, and define the priorities for promoting HA Peers to HA Primary hosts.

Start by navigating to the Agent Properties dashboard, the FglAM namespace, and the FglAMAdapter properties. From there, you can edit the High Availability Host Config list-based property to assign an Agent Manager to HA Partitions and define their eligibility for becoming HA Primary hosts.

To assign an Agent Manager to an HA Partition:

1	Ensure that the Agent Manager is connected to the Management Server.

2	Log in to the Foglight browser interface.

3	On the navigation panel, under Dashboards, navigate to Administration > Agents > Agent Properties.

4	On the Agent Properties dashboard that appears in the display area, in the Namespace/Type view, expand the FglAM node, and click the FglAMAdapter node.

5

Assign the Agent Manager to a desired HA Partition by editing its entry in the High Availability Host Config list. This list contains all Agent Managers that are currently connected to the Adapter, and is accessible through the High Availability Host Config property. The list also identifies the names of their respective HA Partitions, and the priorities for considering Agent Managers as potential HA Primary hosts.

a	Get started with editing the High Availability Host Config list-based property.

In the Properties view, under High Availability, on the right of the High Availability Host Config property, click Edit.

b	Assign the Agent Manager to an HA Partition.

In the dialog box that appears, locate the Agent Manager entry that you want to add to an HA Partition, and in its HA Partition Name column, type the name of that HA Partition. Do not add an Agent Manager to this list if the client you want to assign the HA Partition is not listed here. Agent Managers that connected to this Management Server will be automatically added to this list. Manually adding Agent Mangers is not supported.

To assign more Agent Managers to this or other HA Partitions, repeat this step.

c	Define the priority used to consider this Agent Manager as an HA Primary host.

When promotions to the HA Primary role are evaluated, the FglAM Adapter gives preference to the HA Peer with the highest priority. Acceptable values are in the minus ten to ten range. Use them to decrease or increase the likelihood of an HA Primary role assignment. Type the desired value in the Priority column, or leave the default value of zero.

To define the priorities of other Agent Managers, repeat this step.

d	Click Save Changes.

 

TIP: To get an overview of the HA Configuration in real time (for example, to see the current HA Primary hosts, and the HA State of each HA Peer), you can view the HAManagerMBean. Using the JMX-Console, click FglAM:name=HAManager and invoke the diagnosticSnapshotAsString() method. The resulting output lists each of the known Agent Managers, which (if any) HA Partition they are assigned to, what deployment set they have, who is the HA Primary and what HA State they are in.

Adding cartridges to the HA deployment whitelist

In addition to assigning an Agent Manager to an HA Partition, real HA activities do not take place until HA Aware agent types are deployed.

To provide eligibility for HA agent fail-over to existing agents that primarily function as remote monitors, the HA Deployment Whitelist property is included. This list-based property identifies prior cartridge deployments that were not originally marked as eligible for HA agent fail-over. In this list, each row entry contains a regular expression pattern that matches the agent package, version and agent types that are permitted to be managed in an HA Partition.

Understanding the criteria for HA eligibility

The criteria for inclusion into this list is that the agent type must be capable of monitoring remote resources. It does not make sense to enable HA fail-over for an agent that is collecting data that can only be retrieved from the Agent Manager host on which it is running. If such an agent is added to the Whitelist, when the host fails over, the agent can potentially start monitoring the resources of the newly promoted HA Primary, which can result in unexpected outcome when viewing the dashboards that display the collected metrics.

 

IMPORTANT: To effectively be eligible for fail-over functionality, agents must be using one of the RemoteConnection Services available in the Agent Manager Development Kit.

To add an entry to the Whitelist:

1	Ensure that the Agent Manager is connected to the Management Server.

2	Log in to the Foglight browser interface.

3	On the navigation panel, under Dashboards, navigate to Administration > Agents > Agent Properties.

4	On the Agent Properties dashboard that appears in the display area, in the Namespace/Type view, expand the FglAM node, and click the FglAMAdapter node.

5	Add a new regular expression to select a desired agent type and add it to the Whitelist.

a	Get started with editing the HA Deployment Whitelist property.

In the Properties view, under High Availability, on the right of the HA Deployment Whitelist property, click Edit.

b	Add a new regular expression to the table.

In the dialog box that appears, click Add Row, and in the new row that appears, type a regular expression using the following syntax:

[Agent_package]/[Agent_version]/[Agent_type

 

IMPORTANT: The elements in the expression must be separated with slashes ‘/’.

c	Click Save Changes.

6	If the cartridge containing the specified agent type is not already installed, install it using the Cartridge Inventory dashboard, and deploy its agent package to an Agent Manager that is assigned to an HA partition.

About agent fail-over

When the HA Primary host goes off-line, a schedule is registered that delays the transfer of the agents (and the promotion of another HA Primary) for two minutes. This delay is in place to allow the HA Primary host to be restarted and retain its HA status and assigned agents. If you find that the duration is too long, the following startup parameter can be set when launching the Management Server:

-Dfglam.ha.agent.transfer.delay.milliseconds=XXXX