NetVault regularly hangs with large amount of nvsendopmsg and nvplgnotify processes (79863)

NVGUI cannot be used and the following pop up message may be returned:
"Auditor failed to response to request".

Examination of the system resources show that

• a large amount of 'nvsendopmsg' and nvplgnotify have been created but never exited.
• 'nvgui' and 'nvguidisplay processes' may also fail to start. 
• The nvavp process is consuming a large amount of system memory and this value keeps growing. 
• The ..netvault/db/audit.bin file is growing in size and could reach 500MB

The Auditor is heavily solicited by a surge in notifications events. Each notification requiring an "nvsendopmsg" and "nvplgnotify" process to be spawned and run, the process manager runs out of available process slots to adopt these new processes.

The Auditor Daemon (nvavp) is responsible for tracking, controlling and validating user activities and requests. Depending on the user privileges, the Auditor grants or denies the requests. These events are logged in the .../db/audit.bin file also known as the Audit Trail. You can view these logs via NVBU Audit Trail report. By default they are deleted after 31 days.

You can change the size, content and age of this file by using the nvconfigurator > Auditing tab options. If a large amount of notifications are setup and the events to be notified on happen repeatedly in large occurrences, the process manager will reach a point where there are no more process slots available to adopt the wave of new processes.

To attempt to recover from this situation:

1. Kill spurious nvplgnotify or nvsendopmsg processes using Linux\Unix kill command or Windows Task Manager until they are all terminated
2. Ensure that the licensing is up to date and that any licence parameter is NOT being exceeded
3. Disable Notification of sysopmsg and email for all but essential messages, ideally disable all.
4. Purge the Auditor files by setting a frequent purge in Configurator -> Auditing
   This should reduce the number of messages held and therefore the memory footprint of the Auditor. Also Increase the purge frequency as well as reduce the age/size.
5. Delete the Operator messages regularly via the GUI Status window