Job Manager and Child Process died errors on Windows (4045930)

When many jobs (greater than the default 20, typically 30-60) are configured to run simultaneously, beyond a certain point all further jobs that start to run will instantly fail. When the majority of jobs have finished, subsequent jobs will succeed at running once more without any user intervention.

This is because of a windows limitation in the heap size given to services. When NetVault Backup (NVBU) spawns a new process for a job, the windows CreateProcess API can fail silently meaning that the NVBU process cannot run.

Even after making configuration changes on the Schedule Manager tab from 20 to the required number, the following errors are still received;

Job Manager Died
Child Process died unexpectedly
Fatal error: Duplicate Died Unexpectedly

In NVBU 7.1.1 and before, the NVBU service was given access to a heap of 3MB by virtue of the service being allowed to 'interact with the desktop'. This can represent a security risk, so from version 7.1.2 this is no longer enabled during installation leaving the service with a default heap size of only 512kB (or 768kB on 64-Bit systems).

Therefore the heap size needs to be increased to permit additional jobs to execute simultaneously. This involves editing the Windows registry and increasing the heap available to 'non-interactive' services.

N.B. This is a system-wide setting. Modifying the values incorrectly might lead to a non-bootable system.

The confirmation that this is the error can be achieved very simply. Edit the "Netvault Process Manager" service properties, and on the "Log On" tab allow the service to interact with the desktop. If the errors disappear then untick this box and perform the following long-term workaround, also described in a MS knowledge base article at https://support.microsoft.com/en-us/kb/126962

1. Using a registry editor navigate to:
   \HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\Session Manager\SubSystems.
2. Locate and select the registry key “Windows”.
3. Right click and choose “Modify”.
4. In the “Value data” field find “SharedSection=1024,3072,512” (32-bit Windows example).
   The second component is the memory allocated for "interactive services".
   The third component is the memory allocated for "non-interactive services".
   Increasing this last number to 2048 has been found to allow up to 60 jobs to run concurrently.
5. Change it to:
   “SharedSection=1024,3072,2048” (on 32-Bit Windows)
   “SharedSection=1024,20480,2048” (on 64-Bit Windows)
6. Press OK
7. Reboot the system for the changes to take effect.

On Operating Systems earlier than Windows Vista or Server 2008, there is a maximum of 48MB across these services (configurable through the SessionViewSize value). So if in excess of 20 'non-interactive' services attempt to run, they would have used ~10MB (~20%) and now use ~40MB (~90%) of the total resource available, therefore negatively affecting performance for other services.

Excess services may die silently because windows has exhausted its non-interactive heap. Therefore the workarounds are:

• Modify job scheduling
  You can spread out the jobs so that not so many are queueing at one time, for example by running the quicker jobs first. These finish quickly, so that when the slower jobs are initiated there are less jobs queuing.
  Scheduling jobs later should not be too much of a disadvantage, since they are not running but queuing.
• Registry Edit
  The memory available to non-interactive services system wide can be modified as described above.
• Interact with Desktop
  Enabling this changes both the heap size available to services, but also grants additional privileges. The combination of networked clients and local privileges can represent a security risk.
• Netvault on Linux
  This limitation is windows only, the Netvault server could be run on linux.