OS collection fails with "Too many open file" errors (4229411)

OS collection fails with "Too many open file" errors

The Operating System (OS) connections and metrics fail intermittently in the database dashboard for Oracle agents on Linux.

The hard nofile and soft nofile on the fglam host shows 65536, which is more than enough for the database cartridge.

Other system configuration files, like /etc/profile.d/limits.sh & /etc/security/limits.conf contain no file limitations.

 

The OS connection and metrics collections require opening files on the Foglight Agent Manager host. This number of open files shouldn't be higher than 1000 open files, on average.

There is however a scenario where the OS collections fail with "too many open files", yet the FglAM host settings and configuration - starting with ulimit command - show that there is no restriction on opening files in the system. This can be caused by the FglAM self-start script which controls the number of open files in the system.

        1. Login into the FglAM host with the FglAM user (the user that is used to start/stop FglAM)
        2. Check .bash_history for that user
          - there is a good chance that it will look something like this:
            #1350552720
            /etc/init.d/quest-fglam start
            #1350552973
            /etc/init.d/quest-fglam stop
          - That means that the FglAM runs its own start/stop script from /etc/init.d.
          - The script name is quest-fglam.
        3. Open the script for editing. it will look something like this:
        echo "Starting FglAM: "
          logMessage "notice" "Starting FglAM from $FGLAM_HOME"
        
          # # Make sure we have enough open files.
          ulimit -n 512

The last line, ulimit -n 512 is what limits the files from being opened and hence no OS collections are available.

WORKAROUND

1. Either remark the ulimit line or set it to higher number.
2. Restart the Fglam (Foglight Agent Manager).

STATUS

FGL-14364 "Errors in Foglight's log should be reported as a alarms "submitted for consideration in future release of Foglight.

FGL-14364

The V5.7.5.8 and lower versions of the FglAM use a default open file limit of 512 in the startup script. 

The V5.9.1 and higher versions of the FglAM use a default open file limit of 4096 in the startup script.

The open file limit was increased to 4096 under Enhancement request FAM-6996 in the V5.9.1 FglAM.
FAM-6996: Increase the number of open file descriptors specified by the "fglam-init-script.sh" from 512 to a more generous allowance