Logs show the following errors:
Error 2010/05/15 19:47:18 0 Schedule INF-BACKUP-3 No available ports for message channel listen sockets
Error 2010/05/15 19:47:18 0 Schedule INF-BACKUP-3 No available ports for message channel listen sockets
Error 2010/05/15 19:47:18 0 Schedule INF-BACKUP-3 No available ports for message channel listen sockets
These entries are followed shortly by the following entries in the binary log:
Error 2010/05/15 19:52:20 32974 Jobs INF-BACKUP-3 Scheduler exited unexpectedly, abandoning job
Error 2010/05/15 19:52:20 87535 Jobs INF-BACKUP-3 Scheduler exited unexpectedly, abandoning job
When the Scheduler process dies, a dump file is created reporting the following:
FILESYS 12687 19 24 200532 Failed to open /netvault/nv/etc/foundationmachines.dat error Too many open files
Netvault process has reached open file limit
The netvault Schedule manager processes reaches limit on how many open files it can open at one time. This stops it from opening ports for jobs to run and also stops it opening nessesary file it requires to schedule jobs. The combibnation of this causes the schedule manger to crash which fails any running jobs trying to run.
To increase system wide open file limit:
First, edit /etc/sysctl.conf and set the entry for fs.file-max to 400000 as below
fs.file-max=400000
then type following command to dynamically invoke this
# sysctl –p
Confirm the new value by running the following command
# sysctl -a |grep fs.file-max
To set the individual process limlits Edit /etc/security/limits.conf add following values
* hard nofile 10000
* soft nofile 5000
