-
Title
ERROR: "java.io.IOException: Too many open files" -
Description
a) Foglight Management Server (FMS) server crashes with an exception :
java.io.IOException: Too many open files
at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:145)
at com.sitraka.pas.nexus.comm.ConnectionListenerService.run(ConnectionListenerService.java:220)
at com.sitraka.pas.nexus.comm.ConnectionListenerService.access$000(ConnectionListenerService.java:47)
at com.sitraka.pas.nexus.comm.ConnectionListenerService$1.run(ConnectionListenerService.java:173)or
b) Foglight Agent Manager (FglAM) server crashes or loses connections
2012-07-10 14:03:47.686 ERROR [Quartz-434] com.quest.glue.core.agent.PersistAgentStatesTask - An unexpected error occurred which may cause undesired behavior. You may want to contact Quest Software customer support if you see this error again: Could not write state to /quest/Quest_Software/Foglight_Agent_Manager/state/default/agents/agentproperties.cache
java.io.IOException: Too many open files
at java.io.UnixFileSystem.createFileExclusively(Native Method)
at java.io.File.checkAndCreate(File.java:1704)
at java.io.File.createTempFile(File.java:1792)
at com.quest.glue.core.agent.PersistAgentStatesTask.persistAgentStates(PersistAgentStatesTask.java:173)
at com.quest.glue.core.agent.PersistAgentStatesTask.execute(PersistAgentStatesTask.java:95)
at com.quest.glue.core.scheduler.quartz.QuartzScheduler$ScheduledTaskSequentialJob.execute(QuartzScheduler.java:716)
at org.quartz.core.JobRunShell.run(JobRunShell.java:202)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662) -
Cause
The cause of the Java exception is that the FMS or FgLAM tried to open a new file but the resource limits have been hit for the maximum number of files open for a single process.
This can be due to a resource limit set in the UNIX shell or imposed by the operating system (kernel) limits.
-
Resolution
To increase the resource limits for the :
1) UNIX shell -The setting of resource limits is UNIX shell-dependant (i.e. the commands can vary from shell to shell), but in the Bourne again Shell (BASH) the shell limits are set by using :1a) Checking shell limits :ulimit -aThis will print out each shell limit with its current setting and the maximum allowed by the operating system1b) Set shell limits (again the commands vary depending on the shell) for BASH use :ulimit -hwhich will set all the shell resource limits to the maximum (please note only a privileged user can increase some shell limits - such as memory, but the files limits should changeable by any user).2) The UNIX Operating system (kernel) open file limits -As with the UNIX shell the method setting a the maximum number of open files allowed by a single process varies between operating systems so please check the systems documentation for the appropriate method.For example:Changing the maximum open files on Linux (again not all Linux operating system are the same but the method below is common to many) :To increase the maximum number of open files you need to adjust the value using the proc file system.In the /etc/rc.d/rc.local file make the following changes :>> #Now System is up, Modify kernel parameters for max open files/inodes> #etc..>> if [ -f /proc/sys/kernel/file-max ]; then> echo 16384 >> /proc/sys/kernel/file-max> fi> if [ -f /proc/sys/kernel/inode-max ]; then> echo 24576 >> /proc/sys/kernel/inode-max> fi> if [ -f /proc/sys/kernel/file-nr ]; then> echo 2160 >> /proc/sys/kernel/file-nr> fiNote - make a backup of any file before making changes -
Additional Information
Please refer to your Unix or Linux documentation for the specific method for your Operating System to raise the number of total open file descripts that can be used by the whole system.
Example third-party references include:
https://blogs.oracle.com/elving/entry/too_many_open_files
https://bugzilla.redhat.com/show_bug.cgi?id=116133