SmartDisk Service stopped during backups "Failed to receive msg header from monitor" (4034219)

SmartDisk Service stopped during backups "Failed to receive msg header from monitor"

SmartDisk service stopped during nightly backups.
Upon restarting the SmartDisk Service all seems to go back to normal.
Root cause analysis shows that the percolatormonitor process "Failed to receive msg header from monitor" before the service stopped.

percolatorslave trace shows:
2 SLAVMSGS ??? 17 210222349362 Sent 'Check exited children' message to Monitor
3 PERC-SLV ??? 14 210222349364 Going to select on 6. Time left 120000000 microseconds
3 PERC-SLV ??? 20 210422412853 Timed out waiting for monitor (after select)
0 SLAVMSGS ??? 15 210422412868 Failed to receive msg header from monitor

All SmartDisk processes are started by the PercolatorMonitor service which runs as a privileged user.
Privilege separation is used, hence the PercolatorSlave process which runs as a non-privileged user, is used to execute most of the tasks on behalf of the PercolatorMonitor.
As the inter process communications are done using CAFE messages, the 'slave' and 'monitor' communicate regularly to keep track of the tasks.
While waiting for responses between processes there is a timeout in place between the percolatorslave and the percolatormonitor.
Should the percolatorslave fail to receive a response back from the percolatormonitor within that timeout, the percolatorslave exits.
Without the slave to perform CAFE communications the rest of the processes fail to send/receive messages resulting in the application shuting down.

We have raised Bug 22517 to change this timeout to be more flexible, but if this timeout has been reached on your machine, it is a clear indication
that SmartDisk may be limited in terms resources and could be under heavy load preventing normal communications from happening in time.
The Slave process has been designed to wait for the sent messages for a defined timeout and if no response is received, it exits.
However, that amount of time it waits, although hardcoded, is of course determined by the OS, the machine, load and real life scenario.
Make sure the minimum hardware requirements for SmartDisk have been met and make sure SmartDisk does not share the same machine with other resource intensive applications.

Bg 22517