FMS becomes unresponsive due to number of processes available to user account resulting in unable to create new native thread (4300147)

FMS becomes unresponsive due to number of processes available to user account resulting in unable to create new native thread

FMS becomes unresponsive and/or agents on Foglight Agent Manager (FglAM) stop collecting data. Logs below error:

"java.lang.OutOfMemoryError: unable to create new native thread"

The issue is seen mainly on Linux platform

This happens whenever the JVM asks for a new thread from the OS. If the underlying OS cannot allocate a new native thread, this OutOfMemoryError will be thrown.

This is not a memory issue, but rather an issue where a new native thread can't be created by the OS.

Linux limits the amount of threads that a user can run simultaneously, this can be verified using this command:

ulimit -u
1024 <=== sample output

Thread Usage:

------------------------------
    Peek thread count: 948
    Current thread count: 208
    Daemon thread count: 47
    Total threads started: 392182[...]

Resolution depends on the environment. Below information obtained from this article from Red Hat.

• Check /etc/security/limits.conf and all files in /etc/security/limits.d/ for the current setting of the nproc value
• Extend the nproc value. This can be done in /etc/security/limits.conf or in any file in a file in /etc/security/limits.d/.
• Example of /etc/security/limits.conf:

• In Red Hat Enterprise Linux 6, there's a default setting in /etc/security/limits.d/90-nproc.conf. Change the default limit for all users or add a new limit for the affected user.

# Default limit for number of user's processes to prevent
# accidental fork bombs.
# See rhbz #432903 for reasoning.
*          soft    nproc     1024
testuser   soft    nproc     10240    <== add

Additional information: https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Linux/5/html/Tuning_and_Optimizing_Red_Hat_Enterprise_Linux_for_Oracle_9i_and_10g_Databases/sect-Oracle_9i_and_10g_Tuning_Guide-Setting_Shell_Limits_for_the_Oracle_User-Limiting_Maximum_Number_of_Processes_Available_for_the_Oracle_User.html http://www.mastertheboss.com/jboss-server/jboss-monitoring/how-to-solve-javalangoutofmemoryerror-unable-to-create-new-native-thread