Credential alarms fire every hour (156825)

Credential Management Event alarms fire hourly from both Windows agents (Infrastructure agents) and database agents.

One of these three separate issues may be encountered depending on the cartridge type:

For the IC (Windows Agent) cartridge alarms

Every time the agent establishes a connection to the monitored host during the collection will triggered those WinRM alarms. They are fired by the Foglight Management Serer (FMS) internal service, so there is no rule to turn off to stop the WinRM alarm fundamentally. 

For any Foglight Agent Manager (FglAM) before 5.8.5, there was a problem that the Infrastructure (IC) agent will trigger a cleared WinRM alarm every hour even though the collection with WMI is working The connection will be established and released in every cycle, but as the agent will remember the last successful connection parameter, the agent will not repeat the WinRM attempt if the WMI connection succeeds. In other words, users should only see the WinRM alarm at the very beginning. Upgrading to 5.8.5 addresses this aspect.

For the SQL Server cartridge alarms

The SQL Server agent, from version 5.7.5.1 onwards, is now using Windows Remote Shell command to get the DBSS_Error_Log_List, DBSS_SQL_Server_Error_Log, and DBSS_SQL_Agent_Error_Log collections. Windows Remote Shell Command connection is a different type of connection that also uses WMI/WinRM apart from the usual WindowsInfoConnection that a DBSS agent makes to get the OS Collections. 

The failed Credential Alarms occur when the DBSS agent cannot make the Remote Shell command and this is a valid alarm (when the 4 WinRM + 1 WMI connection attempts fail). WindowsInfoConnection may very well work but the Remote Shell command may fail, triggering the failed credential alarms.

SQL Server agents may be configured to use FALLBACK Error list collection query method, this attempts to make an OS connection using WMI to collect Error Log data, and then follows with a JDBC connection.

If the sufficient permissions have not been applied, credential alarms from each SQL Server agents may fire hourly.

Run the following groovy script to change the Error Log collection for all SQL Server agents from FALLBACK to JDBC

srvConfig             = server["ConfigService"];
srvAgent             = server["AgentService"];

namespace = "DB_SQL_Server"
agentType = "DB_SQL_Server"

def updateASP(primaryASP) {
     primaryASP.setValueByString("ErrorLogListQueryMode" , "JDBC");
     srvConfig.saveConfig(primaryASP);
}

primaryASP = srvConfig.getAgentTypePrimaryAsp(namespace, agentType);
updateASP(primaryASP);
def allAgents = srvAgent.findByAdapterAndType("FglAM", "DB_SQL_Server");
allAgents.each {agent ->
     primaryASP = srvConfig.getAgentInstancePrimaryAsp(namespace, agentType, agent.getId());
     updateASP(primaryASP);
}

return true;

The following script checks the Error Log connection type setting for all SQL Server agents

def out = new StringBuilder();
def agentService = server.get("AgentService"); def configService = server.get("ConfigService"); def agents = agentService.findByAdapterAndType("FglAM", "DB_SQL_Server"); for (agent in agents) {
   def primaryASP = configService.getAgentInstancePrimaryAsp(agent.getAgentNamespace(), agent.getTypeId(), agent.getId());
   if (primaryASP) {
      out.append(String.format("%s, %s: ErrorLogListQueryMode=%s\n", agent.getRemoteClientId(), agent.getName(), primaryASP.getString("ErrorLogListQueryMode")));
   }
}

return out.toString();

For DB2 and Oracle cartridge alarms

Please refer to KB 186315

Windows registry changes are often necessary on the monitored Windows server for WMI connections from Linux-based FglAMs to be successful. WinRM is the preferred connection mechanism for Windows server monitoring.

Additional CRs: FGL-19666, SSDB-14323, SSDB-14324, SSDB-14325

Idea/Enhancement FGSS-I-32 was logged to ask that the "Error list collection Query Method" property value be set to JDBC by default.