Foglight becomes unresponsive almost immediately after startup (4224000)

Foglight Management Server (FMS) becomes unresponsive almost immediately after startup

Symptoms

• Multiple thread dumps shows eventual JVM heap exhaustion
  • The runnable threads show LDAP computations 

  •     Thread: http-0.0.0.0-8080-7, daemon, id=162, priority=5, state=RUNNABLE, thread group=jboss
            java.util.ArrayList.size(ArrayList.java:177)
            java.util.AbstractList$Itr.hasNext(AbstractList.java:339)
            com.quest.nitro.service.security.core.NitroSecurityService.getAllParentGroupIdsInCache(NitroSecurityService.java:1620)
            com.quest.nitro.service.security.core.NitroSecurityService.getAllParentGroupIds(NitroSecurityService.java:1654)
            com.quest.nitro.service.security.core.NitroSecurityService.getAllParentGroupIds(NitroSecurityService.java:1575)
            com.quest.nitro.service.security.core.NitroSecurityService.getAllRolesForUser(NitroSecurityService.java:1784)
            com.quest.nitro.service.security.auth.spi.NitroExtendedLdapLoginModule.assignAllRolesToUser(NitroExtendedLdapLoginModule.java:1701)
            com.quest.nitro.service.security.auth.spi.NitroExtendedLdapLoginModule.createLdapInitContext(NitroExtendedLdapLoginModule.java:617)
            com.quest.nitro.service.security.auth.spi.NitroExtendedLdapLoginModule.validatePassword(NitroExtendedLdapLoginModule.java:237)

• Management server logs may eventually mention JVM heap exhaustion, but not immediately
• After some uptime, datasource thread dumps eventually generate in the FMS\logs directory (In this case, the FMS repository was SQL server)
  • Exception message: Could not create connection; - nested throwable: (com.microsoft.sqlserver.jdbc.SQLServerException: The driver could not establish a secure connection to SQL Server by using Secure Sockets Layer (SSL) encryption. Error: "Connection reset by peer: socket write error".); - nested throwable: (org.jboss.resource.JBossResourceException: Could not create connection; - nested throwable: (com.microsoft.sqlserver.jdbc.SQLServerException: The driver could not establish a secure connection to SQL Server by using Secure Sockets Layer (SSL) encryption. Error: "Connection reset by peer: socket write error".))
    Thread: Thread[DataAccess-7-thread-1,5,Forge Core Threads]
• Diagnostic Snapshot from the FMS support bundle reveals a "ThreadDeath" exception

  • ---- com.quest.nitro:service=NitroSecurityService

    …

    --------------- LDAP configuration -----------------

    …---------------------- Users -----------------------

    SecUser[userId: <id>, userName: foglight, isExternal: true, isService: false, statusDesc: , badPwdCnt: 0, badPwdTime: Fri Jan 01 00:00:00 EST 1971, createdTS: Tue Mar 04 09:03:29 EST 2014, lastLogonTS: Fri Jan 01 00:00:00 EST 1971, lockoutTS: Fri Jan 01 00:00:00 EST 1971, logonCount: 1, modifiedTS: Fri Jan 01 00:00:00 EST 1971, pwdExpiryTS: Fri Jan 01 00:00:00 EST 1971, pwdLastSetTS: Fri Jan 01 00:00:00 EST 1971, status: A]

    +++ SecGroup[groupId:…

    +++ SecGroup[groupId:…

    Exception in diagnosticSnapshot: java.lang.ThreadDeath

Excessive computations perfomed by Foglight's NitroSecurityService to generate diagnostic info when cyclic LDAP groups are defined

FGL-15889 has been identified and has been resolved in Foglight version 5.6.11 or Foglight for Virtualization 7.2

WORKAROUND 1

• Patch to Foglight 5.6.11 or Foglight for Virtualization 7.2

WORKAROUND 2

• Attached  is a groovy script to remove the excessive LDAP groups
  • Run the groovy script from the FMS Host command prompt
  • Depending if  the fms running on windows (fglcmd.bat) or unix/linux (fglcmd.sh)
  • Example. fglcmd.sh -usr <FMS Admin> -pwd <FMS Admin_pass> -cmd script:run - f <path>/delete-all-external-groups.groovy
    • Note:  An initial Foglight restart may be required to run the steps above as JDBC thread may be exhausted.