Investigating and resolving OS (Operating System) credential issues (4226257)

Investigating OS connection issues can be challenging when there are numerous credentials present in the FMS. This knowledgebase article describes some techniques to identify and resolve common OS credentials issues for Windows and Linux / Unix.

Here are some steps that can be used to identify and fix OS credentials issues in Foglight

1). When a credential is not working, adding credentials lower in relative order (100, 200... 1500, 1600) will not help. The remote host may lock out the password before these are reached or there may be a maximum tries in other credentials. Credentials are reviewed and matched using the relative order. Agents are checked agents the resource mappings in credential in position #100, then #200, etc. 

2). Type the hostname in the search field and look for multiple credentials for the same host. For best results, there should only be one on Windows, and maybe two (of different types) in Linux. If there more credentials, remove the extras and focus on the main credential you want to use for that host.

3). Multiple credentials with the same name can be confusing to manage. Renaming the credential to match the account makes it easier to identify similarly named credentials. It is a best practice to include the domain and user in the credential name.

To change the credential name, select the edit icon next to the credential name and select Credential Name

Replace the credential name with the new name and then click the Save button

Here is an example of how the updated credentials can look

4). Reenter the password. Often the password is wrong, especially if there are alarms for many agents. The password can be changed by click the pencil icon in the credential row, selecting Credential Properties, then by updating the Password and Confirm Password fields. Then click the Save button.

5). Using the FQDN (Fully Qualified Domain Name) is almost always more effective for WinRM krb handling when Windows monitoring than using only the domain name. In the credential properties, change the domain name to use the FQDN for the network environment (e.g. from CORP to CORP.MYCOMPANY.COM)

5). For remote monitoring Windows hosts with WinRM, don't use "Use Client's Login At Connection Time" or "Local Account for Monitoring Windows OS". WinRM does not work with remote WinRM monitoring. Edit the credential by clicking the pencil icon, select Resource Mappings, remove the resource mapping specific to the remote hosts by clicking the remove icon, and then click the Save button.

6). If you are confident one of the credentials is correct for the host, move to 100 in the relative order at the top of the list by selecting the credential checkbox, clicking the Reorder button in the menu, clicking the up arrow in the popup to move the credential to the top of the list, and then clicking the Confirm button. This will make the credential the first credential to be checked when evaluating for use with the agent. 

7). If you still cannot get the credential to connect for a host, delete the resource mappings in all of the credentials referencing that host. Then create a new credential only for that host. Move it to the top of the relative order. This credential can be created manually (e.g. Knowledgebase article 104508) or using the wizards included with the database or infrastructure agents (e.g. validate connectivity as described in Knowledgebase article 104508). The new credential may be created using the name username (or domain/user) specified in the wizard. Moving the new credential to the top of the relative order may help (as in Step 4 above) may help in testing efforts. 

8). Re-Release the lockboxes to the FgLAMs when you are done working on the credentials by Navigating to Administration | Credentials | Manage Lockboxes, selecting the Lockbox checkbox, clicking the Release to Credential Clients icon for the lockbox. In the popup, one or all of the FglAMs listed and click the Release button.

9). If numerous changes have been made to update and correct credentials, yet there still being passed to the agent then rebuilding the credential cache on the Foglight Agent Manager (FglAM) may be beneficial. Some credentials alarms may fire briefly while the agents attempt to reconnect after restarting the FglAM and before the lockboxes have been rereleased to the FglAM.

1. Connect to to FglAM via RDP or SSH
2. Stop the FglAM service
3. Rename the credentials folder {fglam}\state\default\credentials to {fglam}\state\default\credentials_old
4. Restart the FglAM
5. Re-release the lockboxes as described in Step 8 above.

Once a working set of credentials have been completed, it is a best practice to clean up and merge credentials into the minimum set of credentials that are actually needed. Knowledgebase article 338554 describes how to merge credentials.

Troubleshooting issues with many credentials all with the same names and/or user-passwords is challenging and likely to be a problem then the user eventually changes passwords on the hosts.