FMS crashes continuously after upgrading to 7.1 (4375204)

After upgrading Foglight to version 7.1, the FMS fails to start generating a crash dump file like:

C:\Quest\Foglight\hs_err_pid11548.log

The FMS crashes once the login page is displayed.

This Windows server has SentinelOne client installed.

The crash dump file shows messages like these:

#
# A fatal error has been detected by the Java Runtime Environment:
#
#  EXCEPTION_ACCESS_VIOLATION (0xc0000005) at pc=0x000000000d6bd533, pid=11548, tid=10548
#
# JRE version: OpenJDK Runtime Environment Zulu17.44+15-CA (17.0.8+7) (build 17.0.8+7-LTS)
# Java VM: OpenJDK 64-Bit Server VM Zulu17.44+15-CA (17.0.8+7-LTS, mixed mode, sharing, tiered, compressed class ptrs, g1 gc, windows-amd64)
# Problematic frame:
# J 35887 c1 javax.servlet.http.HttpServletRequestWrapper._getHttpServletRequest()Ljavax/servlet/http/HttpServletRequest; (8 bytes) @ 0x000000000d6bd533 [0x000000000d6bd4a0+0x0000000000000093]
#
# Core dump will be written. Default location: C:\Quest\Foglight\hs_err_pid11548.mdmp
#
# If you would like to submit a bug report, please visit:
#   http://www.azul.com/support/
#

---------------  S U M M A R Y ------------

Command Line: -dsa -da -Xrs exit -Dquest.state.dir=C:\Quest\Foglight\state -Xms32768m -Xmx32768m -Djava.awt.headless=true -Dsun.rmi.dgc.client.gcInterval=86400000 -Dsun.rmi.dgc.server.gcInterval=86400000 -XX:+DisableExplicitGC -XX:CompressedClassSpaceSize=512m -XX:ReservedCodeCacheSize=256m -XX:+ForceTimeHighResolution -XX:+UseG1GC -XX:+UseStringDeduplication -Djava.security.properties==C:\Quest\Foglight\config\security\java.windows.security -Dsun.lang.ClassLoader.allowArraySyntax=true -Djava.security.manager=com.quest.nitro.bootstrap.ForgeSecurityManager -Djava.security.policy=C:\Quest\Foglight\config\server.policy -Djavax.management.builder.initial=com.quest.nitro.bootstrap.ForgeMBeanServerBuilder -Djmx.invoke.getters --add-opens=java.base/java.io=ALL-UNNAMED --add-opens=java.base/java.lang=ALL-UNNAMED --add-opens=java.base/java.nio.charset=ALL-UNNAMED --add-opens=java.base/sun.net.util=ALL-UNNAMED --add-opens=java.base/sun.security.internal.spec=ALL-UNNAMED --add-opens=java.desktop/sun.font=ALL-UNNAMED --add-opens=java.management/com.sun.jmx.mbeanserver=ALL-UNNAMED --add-opens=java.security.jgss/sun.security.krb5=ALL-UNNAMED --add-opens=jdk.management/com.sun.management.internal=ALL-UNNAMED -Dquest.common.process-runner.disabled=true -Dquest.saml.hostname=hostname01.example.com -Dfoglight.credential.alarms.disabled=true -Djava.library.path=C:\Quest\Foglight\jre\bin\server;C:\Quest\Foglight\jre\bin;C:\Windows\system32;C:\Windows;C:\Windows\System32\Wbem;C:\Windows\System32\WindowsPowerShell\v1.0\;C:\Windows\system32\config\systemprofile\AppData\Local\Microsoft\WindowsApps;C:\Quest\Foglight\bin\ -Dquest.common.process-runner=false -agentpath:C:\Program Files\SentinelOne\Sentinel Agent 23.2.3.358\SentinelJava64.dll=extendedCapabilities com.quest.nitro.bootstrap.ForgeMain

Host: Intel(R) Xeon(R) Platinum 8171M CPU @ 2.60GHz, 16 cores, 127G,  Windows Server 2016 , 64 bit Build 14393 (10.0.14393.5786)
Time: Tue Mar 19 12:45:14 2024 Coordinated Universal Time elapsed time: 222.390635 seconds (0d 0h 3m 42s)

---------------  T H R E A D  ---------------

Current thread (0x0000000034037230):  JavaThread "http-exec-3" daemon [_thread_in_Java, id=10548, stack(0x0000000037de0000,0x0000000037ee0000)]

Stack: [0x0000000037de0000,0x0000000037ee0000],  sp=0x0000000037edb700,  free space=1005k
Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code)
C  0x000000000d6bd533


siginfo: EXCEPTION_ACCESS_VIOLATION (0xc0000005), reading address 0xffffffffffffffff

Other dumps show a different problematic frame

# Problematic frame:
# V  [jvm.dll+0x5657b1]

From the crash dump, we can see the SentinelOne client is injecting itself an agent into the JVM at the moment it starts, and some of its resources being called during startup.

Command Line: -dsa -da -Xrs exit -Dquest.state.dir=C:\Quest\Foglight\state [...] -agentpath:C:\Program Files\SentinelOne\Sentinel Agent 23.2.3.358\SentinelJava64.dll=extendedCapabilities com.quest.nitro.bootstrap.ForgeMain
[...]
Dynamic libraries:
[...]
0x00007ffbc2440000 - 0x00007ffbc2763000     C:\Program Files\SentinelOne\Sentinel Agent 23.2.3.358\InProcessClient64.dll
[...]
0x00007ffb93cd0000 - 0x00007ffb93d7d000     C:\Program Files\SentinelOne\Sentinel Agent 23.2.3.358\SentinelJava64.dll
[...]
dbghelp: loaded successfully - version: 4.0.5 - missing functions: none
symbol engine: initialized successfully - sym options: 0x614 - pdb path: .;C:\Quest\Foglight\bin;C:\Windows\SYSTEM32;C:\Program Files\SentinelOne\Sentinel Agent 23.2.3.358;C:\Quest\Foglight\jre\bin\server;C:\Quest\Foglight\jre\bin

VM Arguments:
jvm_args: -dsa -da -Xrs exit -Dquest.state.dir=C:\Quest\Foglight\state [...] -agentpath:C:\Program Files\SentinelOne\Sentinel Agent 23.2.3.358\SentinelJava64.dll=extendedCapabilities 

This resolution is also discussed in KB 4370250 specifically for FglAM (Agent Manager).

To resolve the issue it is required that users exclude directories from being scanned by SentinelOne protection tools, these are common installations directories:

But installations could be located in a different directory.

SentinelOne has an Exclusion Mode called Interoperability, which we have found resolves this issue when applied.

Once [Interoperability] exclusions are in place, restart the Foglight service