NetVault service fails to start with "Invalid programs file" "Failed to get IPC key" (4039549)

NetVault service fails to start with "Invalid programs file" "Failed to get IPC key"

Following a reboot of a Linux NetVault Backup server, the NetVault service won't start.
Trace on the nvpmgr process (NetVault Process Manager) shows it fails to allocate shared memory from the system as the 'programs.cfg' file is invalid.

----------
3   PMGR   ???   1815   25   0   122821716553   PmgrInitSharedMem
3   PMGR   ???   1815   91   0   122821716750   lShmSize 524288, default 512000
5   SHM   ???   1815   6   0   122821976183   ShmAllocate(3f1d0, 524288)
5   IPC   ???   1815   1   0   122821976192   Locating path for IPC key generation
6   FILE   ???   1815   15   0   122821976234   FileStreamOpen(6ca08, /opt/netvault/config/programs.cfg, 0x101)
5   IPC   ???   1815   2   1   122821977161   Invalid programs file?
5   IPC   ???   1815   3   1   122821977169   Failed to get IPC key (0,0)
0   PMGR   ???   1815   27   0   122821977190   failed to get shared memory for process manager
6   SYSTEM   ???   1815   20   0   122821977213   Logging 'Process 'nvpmgr' failed to get shared memory' to system logs
0   PMGR   ???   1815   158   0   122821977410   Failed to initialize shared memory
--------

When nvpmgr starts, it allocates shared memory from the system so it can start the rest of the processes and communicate with them.
To know how much shared memory is needed the following 2 files are read:
/usr/netvault/config/programs.cfg
/usr/netvault/config/procmgr.cfg
In our example, the installation reads that we need to allocate 524288 bytes of shared memory when we need 512000: lShmSize 524288, default 512000
On a working installation the call should normally be by default for 512000 bytes for Unix/Linux: lShmSize 512000, default 512000
The files in question contain invalid values causing the startup to fail.
Configuration files corruption is possible when space on the partition hosting them is exhausted or while a commit on a file occurs while the system is unexpectedly shutdown. 

In order to troubleshoot the issue further:
- Collect trace on a failed NetVault startup for review by Technical Support.
- If the issue is diagnosed to be with corrupt configuration files, Support will provide you with new default files which can safely be replaced and allow for the service to start.
- Rename the old files or move them out of /usr/netvault/?
- then copy the files supplied by Support into /usr/netvault/config
- Start NetVault which should remain started

In the event replacing the above config files does not help, you may consider re-installing NetVault to get a default set of config files from a fresh installation,
A restore of the last NVDB backup taken before the problem, will be necessary to go back to the point in time before the issue occurred.

If you opt for this option, in order to make the process easier, perform the following before re-installing:
- Keep a copy of the below 2 files
/opt/netvault/db/MediaDatabase/Indexes/Devicetree
/opt/netvault/db/MediaDatabase/Records/Devicetree
- Make note of the tape label and location of the media containing the last NVDB restore.
- Once you have re-installed NetVault, stop the netvault service and copy the 2 DeviceTree files mentioned above, to their original location overwriting the new ones.
- Upon restarting netvault you should have all your devices back and all that is left to do is scan the tape that has the NVDB to restore, and restore it without the config files.