Error: "java.net.SocketException: Too many open files" (4324502)

On a number of occasions the Foglight Management Server (FMS) or Agent Manager (FglAM) seems to be in a hung state and it is not possible to login to the UI. The only way to get the FMS or FglAM to respond again is to restart. The system then stays stable for a couple of days then the symptoms return. The following error occurs in the FMS logs -

WARN [ServerSocketListener] jacorb.iiop.listener - java.net.SocketException: Too many open files

Unix kernel parameter for open files is set too low

From the root prompt get the output of ulimit -a

$ ulimit -a
core file size (blocks, -c) 0
data seg size (kbytes, -d) unlimited
scheduling priority (-e) 0
file size (blocks, -f) unlimited
pending signals (-i) 60289
max locked memory (kbytes, -l) 32
max memory size (kbytes, -m) unlimited
open files (-n) 1024
pipe size (512 bytes, -p) 8
POSIX message queues (bytes, -q) 819200
real-time priority (-r) 0
stack size (kbytes, -s) 10240
cpu time (seconds, -t) unlimited
max user processes (-u) 60289
virtual memory (kbytes, -v) unlimited
file locks (-x) unlimited


Check the number of open files, which is 1024 in this instance. Increase this value to 8192

See the following articles for further information:

http://www.cyberciti.biz/faq/making-changes-to-proc-filesystem-permanently/

http://linux-quirks.blogspot.ca/2011/02/too-many-open-files.html