-
Title
Steps to change the password of the Foglight agent account without locking out the SQL Server agents (using Connection Details - Verify Connection) -
Description
One or more FMS systems are monitoring Windows servers using SQL Server agents, HostAgents (Infrastructure) and/or Oracle agents.
The logins share a common Active Directory (AD) account for the Windows and database instance logins.
When remote monitoring is enabled, the shared AD service account quickly becomes locked for all of the agents. This can affect hundreds or thousands of agents.
This is most common after a password change in Active Directory so Foglight monitoring agents may still use the old password.
How can a user find out which agent is causing the lockout?
-
Cause
When a common password is used for multiple FMS systems, then the password change must be done for all agents sharing the same AD account at the same time.
Collections are done every second and with hundreds of agents there can be a millisecond of difference between each agent trying to make the connections.
If the password change is not managed, then any single account can lock out the account because agents run constantly hitting the AD account.
As the number of agents using a single AD credential increases, the more critical it is to
- keep the credentials organized,
- there are numerous credentials in the same FMS that share the same account and reference the same agents multiple times in several resource mappings
- disable all of the agents when updating password
- keep the credentials organized,
-
Resolution
The quickest means to prevent account lockups is to stop the OS account from being use and reset the agents and credentials. As there isn't a way to easily determine which agent is causing the lockout because the logs will quickly fill up with error messages, this is a process to clear all of the old credentials and begin with a clean empty lockbox.
An ALTERNATE set of steps can be followed as described in KB 4308764 use the Agent Status and credentials dashboards to reset the passwords.- Then by using the connection validation, it would create a new single credential in the DB-Lockbox with the new password and resource mappings.
- This one is helpful if there are several credentials in the lockboxes all with different resource mappings, or multiple credentials with duplicate resource mappings.
- Navigate to Administration | Agents | Agent Status
- Stop all database and infrastructure agents (Windows and Unix) (any agents that are using that user ID to connect to the host)
- Navigate to Administration | Credentials | Manage Credentials
- Delete all credentials in lock-box except for System Local Account credentials, make sure there are no hostname resource mappings in these two credentials
- Return to Global View databases dashboard
- Unlock account in Active Directory (AD)
- Next validate the connection for ONE database agent with OS monitoring as described in KB 4235896 (for SQL Server) and KB 4235896 (for Oracle).
- If the validation for that agent is successful, repeat the validation for the other database agents and Save Changes.
- Navigate to Administration | Credentials | Manage Credentials
- Confirm that there is only one new credential in the DB-Agent lock-box.
- Navigate to Administration | Agents | Agent Status
- Reactivate the database agent and windows agents again.