Troubleshooting WinRM host monitoring (4217338)

Steps to aid in configuring and troubleshooting a Windows host for remote monitoring via WinRM (Windows Remote Management)

After following KB 4295397 and configuring WinRM using "winrm quickconfig", what additional troubleshooting steps can be followed?

Identifying Common Causes of WinRM Failures

The following conditions can result in a WinRM failure:

• The WinRM listener is not configured on the remote machine.
• Blocked connections from the Foglight Agent Manager host to the WinRM listener ports on the remote machine.
• The AllowUnencrypted WinRM service configuration setting is false, and HTTP is used on the remote machine.
• The expected authentication type is not enabled for the WinRM service on remote machine. For example, Basic authentication is expected to be used, but it is not enabled.
• A fully qualified domain name is not specified in the user credential created on the Management Server.
• An incorrect Kerberos configuration file is used. For example, the file does not exist on a UNIX system, or the realm is not set up for cross-realm authentication.
• Key Distribution Center (KDC) host configured in the Kerberos configuration file is not reachable on port 88 from the Foglight Agent Manager host.
• The WinRM parameter TrustedHosts is not set.

Please refer to the latest version of the Foglight Agent Manager user guide for specific details on Configuring Windows Remote Management (WinRM).

Steps to investigate from Foglight Management Server (FMS)

1. Navigate to: Administration > Credentials > Manage Credentials
2. Select the credential to be used for monitoring of the remote host and click to edit then Credential Properties
   1. If using a domain service account for remote monitoring, ensure the following is performed
      1. Insert the Fully Qualified Domain under the Domain field

         Example: yourdomain.com

      2. Insert the User Name without a domain under the User Name field

         Example: foglight_svc

   2. ​If using a local service account for remote monitoring, ensure the following is performed
      1. ​Leave the Domain field empty
      2. Insert the User Name without a domain under the User Name field

         Example: foglight_svc

Steps to investigate from the Foglight Agent Manager (FglAM)

1. Verify the Kerberos configuration file (krb5.config). Refer to section Generating a configuration file required for WinRM Negotiate authentication of the Foglight Agent Manager Guide.
2. Verify that the Monitored Host is reachable on port 5985 if using WinRM over HTTP or port 5986 if using WinRM over HTTPS.
3. Verify that the KDC (Key Distribution Center) configured in the Kerberos configuration file is reachable on port 88.

Steps to investigate from the Monitored Host

Verify the WinRM service configuration

1. Remote to target host and open command line with admin privileges
2. Run the following command winrm get winrm/config to display the WinRM configuration for this host
3. Make note of the following items under "Service"

   Example:

   Service
       RootSDDL = O:NSG:BAD:P(A;;GA;;;BA)(A;;GR;;;IU)S:P(AU;FA;GA;;;WD)(AU;SA;GXGW;;;WD)
       MaxConcurrentOperations = 4294967295
       MaxConcurrentOperationsPerUser = 1500
       EnumerationTimeoutms = 240000
       MaxConnections = 300
       MaxPacketRetrievalTimeSeconds = 120
       AllowUnencrypted = false
       Auth
           Basic = false
           Kerberos = true
           Negotiate = true
           Certificate = false
           CredSSP = false
           CbtHardeningLevel = Relaxed 

4. If the WinRM configuration does not display due to it not being configured, run winrm quickconfig or the abbreviated version winrm qc to configure.

Configuring the remote host for monitoring with WinRM

From the WinRM service configuration output:

1. Using a domain service account
   1. Ensure AllowUnencrypted = true; set with winrm set winrm/config/service @{AllowUnencrypted="true"}
   2. Ensure Basic = false; set with ​winrm set winrm/config/service/auth @{Basic="false"}
   3. Ensure Kerberos = true; set with winrm set winrm/config/service/auth @{Kerberos="true"}
   4. Ensure Negotiate = true; set with winrm set winrm/config/service/auth @{Negotiate="true"}
2. Using a local service account
   1. Ensure AllowUnencrypted = true; set with winrm set winrm/config/service @{AllowUnencrypted="true"}
   2. Ensure Basic = true; set with ​winrm set winrm/config/service/auth @{Basic="true"}
   3. Ensure Negotiate = true; set with winrm set winrm/config/service/auth @{Negotiate="true"}

Notes:

• The AllowUnencrypted property must be set to true when establishing WinRM connections over HTTP; if set to false additional configurations need to be completed for connections to be established over HTTPS; for more information refer to How to configure WinRM HTTPS using a certificate? (4327838).
• Basic authentication is used when using a local service account.
• For some environments the WinRM service configuration may be controlled by a Group Policy Object (GPO).

Additional troubleshooting steps

Reviewing Application Event Logs

WinRM logs activity to an event log on the target machine. This includes both success and failure messages for authentication.

To view application event logs:

1. On the target machine, right-click My Computer and select Manage.
2. In the navigation tree on the left, choose System Tools | Event Viewer | Applications and Services Logs | Microsoft | Windows | Windows Remote Management | Operational.
   The default Operational log contains the most common events.

To enable additional debug logging information:

1. Click View.
2. Click Show Analytic and Debug Logs.
3. Right-click the log file you want to view.
4. Select Enable Log.

Enabling connection type debugging from the Foglight Agent Manager (FglAM)

If the only information you are interested in is the types of connections that are being established, there is a command-line setting that enables logging the connection types.

Run the Agent Manager with the switch -Dquest.debug.windowsinfo.types.

Note: This logging occurs every time a connection is established and can be very verbose. It is recommended for debugging purposes only.

For additional debug logging refer to: How to enable Kerberos Debug at the FglAM level (4309953).

When investigating WinRM configuration problems please collect the following items

1. Support Bundles for the Foglight Management Server (FMS) and Foglight Agent Manager (FglAM) as per How to generate Support Bundles, enable debug logging, and Thread Dumps for FMS and FGLAM (4302611)
2. Output of the WinRM configuration
   winrm get winrm/config >> _winrm.cfg
3. Output of the WinRM listener configuration
   winrm enum winrm/config/listener >> _winrmlistener.cfg
4. Export of historical "Credential Management Event" alarms as per How to export alarm history from the Alarms dashboard (4226150)