Hostname switches between two SQL Server hosts triggers false DBSS Cluster Failover alarms (4378866)

Hostname switches between two SQL Server hosts triggering false DBSS Cluster Failover alarms

Example alarm

Foglight has generated a Warning alarm for AGENTNAME : DBSS - Cluster Failover Rule: DBSS - Cluster FailoverSeverity: Warning
Message: Cluster Failover. The SQL Server cluster group active node has moved from node SERVER1 to node SERVER2 Created on: 2025-01-16T06:11:42-08:00 

Messages like the following appear in the SQL Server agent log file

2025-01-16 05:11:05.411 INFO    [AGENTNAME-lowPriorityPool-2-[DBSS_Host_Group][2025/01/16 05:11:05]] com.quest.qsi.fason.sqlserver.agent.SqlServerAgentData - Physical host name changed from SERVER1 to SERVER2.
2025-01-16 05:45:13.549 INFO    [AGENTNAME-lowPriorityPool-2-[DBSS_Maintenance][2025/01/16 05:45:13]] com.quest.qsi.fason.sqlserver.agent.SqlServerAgentData - Physical host name changed from SERVER2 to SERVER1.
2025-01-16 06:01:54.386 INFO    [AGENTNAME-lowPriorityPool-1-[DBSS_Maintenance][2025/01/16 06:01:54]] com.quest.qsi.fason.sqlserver.agent.SqlServerAgentData - Physical host name changed from SERVER1 to SERVER2.

When two different SQL Server agents are creating a JDBC connection at the same time (multiple threads) the second agent is getting the same connection object from the JDBC driver.

Imagine two people trying to use the same telephone at the same time.

• SQL Sever Agent A picks up the phone (JDBC connection)
• SQL Server Agent B also picks up the same phone because the phone system thinks it’s okay to share it

Both agents now talk over each other, which leads to:

• Wrong metrics
• Query failures
• Agent warnings or timeouts

This happens because the JDBC driver or connection pool reuses connections when two threads request a connection at the same time. Instead of issuing two unique database sessions, it returns the same one.

Before the fix:

• Two Foglight SQL Server agents asked the JDBC driver for a connection at the same time
• The connection code was not synchronized, so both agents could receive the same connection object

After the fix:

• The connection creation code was updated to synchronize access, meaning only one agent can create a connection at a time

Workaround

1. Stop the FglAM (Foglight Agent Manager)
2. Download the attached hotfix file from this article. Users must be logged into Supportlink to download the file.
3. Open {FglAM location}\agents\DB_SQL_Server\{version}\lib
4. Move the existing connections-jdbc-{version}.jar and connections-mssql-{version}.jar files to different folder for backup reasons
5. Copy the newer attached connections-jdbc-{version}.jar and connections-mssql-{version}.jar files into the {FglAM location}\agents\DB_SQL_Server\{cartridge version}\lib folder
6. Restart the FglAM 

Note: 

• HFIX-530.zip should work for SQL Server cartridges 6.3 through 7.2.3. 
• HFIX-537 should work for SQL Server cartridge on 7.3.0.x releases.
• Contact Support for a version 8.0 release of the hotfix. 

Status

This issue has been logged as FOG-11777 and will be fixed in an upcoming release of the SQL Server cartridge.