NetVault becomes unresponsive with hundreds of nvsendopsmsg and nvplgnotify processes (4042582)

Date - June 2010
Affected Product & Version - NetVault 8.2.x and 8.5.x
Affected Module & Version - N/A
OS Version - All
Application Information - N/A

Symptoms:
NetVault becomes unresponsive.
NVGUI cannot open and any new nv*** process that tries to spawn will die with:
"Failed to get adopted by NetVault"

ps -ef | grep nv shows hundreds if not thousands of the following lines:

root 27514 0.0 0.0 133332 1904 ? S 01:59 0:00 \_ nvplgnotify 8172
root 27515 0.0 0.0 63804 1052 ? S 01:59 0:00 | \_ /bin/sh /usr/netvault/scripts/global/sysopmessage.sh
root 27516 0.0 0.0 7072 2956 ? S 01:59 0:00 | \_ /usr/netvault/util/nvsendopmsg -n

Reasons:
The Auditor is heavily solicited by a surge in notifications events.
Each notification requiring an "nvsendopmsg" and "nvplgnotify" process to be spawned and run, the process manager runs out of available process slots to adopt these new processes.

Explanations:
The Auditor Daemon (nvavp) is responsible for tracking, controlling and validating user activities and requests.
Depending on the user privileges, the Auditor grants or denies the requests.
These events are logged in the .../db/audit.bin file also known as the Audit Trail.
You can view these logs via NVBU Audit Trail report.
By default they are deleted after 31 days.
You can change the size, content and age of this file by using the nvconfigurator > Auditing tab options.
If a large amount of notifications are setup and the events to be notified on happen repeatedly in large occurrences,
the process manager will reach a point where there are no more process slots available to adopt the wave of new processes.

In an attempt to recover from this situation:
1. Kill spurious nvplgnotfy or nvsendopmsg processes using kill -9 or Task Manager until they are all terminated
2. Ensure that the licensing is up to date and that any licence parameter is NOT being exceeded
3. Disable Notification of sysopmsg and email for all but essential messages, ideally disable all.
4. Purge the Auditor files by setting a frequent purge in Configurator -> Auditing
This should reduce the number of messages held and therefore the memory footprint of the Auditor.
Also Increase the purge frequency as well as reduce the age/size.
5. Delete the Operator messages regularly via the GUI Status window

Also see KB 5455