-
Title
Failed to update RAS device information in the database -
Description
All Data Copy jobs from DR to tape are failing with:
----------------
MSDataCopy Master: Slave [1109] still running error for Channel Down
Process 'Data Copy Plugin Slave' (PID '1109') has crashed!
Process 'RAS Controller' (PID '144') has crashed!
Failed to update RAS device information in the database
----------------
The Data Copy and RAS Controller processes leave dump files showing:
---------------6 DRLAYER ??? 2288 34 0 200044781416 block size is 5242880 MEM ??? 2288 17 12 200044781442 Failed to allocate 524308 byte: The access code is invalid.0 MEM ??? 2288 17 12 200044781451 Failed to allocate 524308 byte: The access code is invalid.0 MEM ??? 2288 4 1 200044781453 Unable to allocate 524288 bytes of memory0 PROCESS ??? 2288 209 0 200044781476 Crash program----------------Some DR containers are offline while others on the same device are online.--------------------------------------------Log Name: SystemSource: Microsoft-Windows-Resource-Exhaustion-DetectorDate: 12/10/2016 20:57:43Event ID: 2004Task Category: Resource Exhaustion Diagnosis EventsLevel: WarningKeywords: Events related to exhaustion of system commit limit (virtual memory).User: SYSTEMComputer: BACKUPDescription:Windows successfully diagnosed a low virtual memory condition. The following programs consumed the most virtual memory: StoreHost64.exe (5760) consumed 11546939392 bytes, elasticsearch-service-x64.exe (1332) consumed 2336055296 bytes, and nvmedmgr.exe (392) consumed 363335680 bytes.----------------------------- -
Cause
The server's Virtual Memory was exhausted due to the StoreHost64.exe process.
This is a Recovery Manager for Exchange process which is used during restores of mail items from Exchange .edb and log files previously restored by NetVault.
Having exhausted the Virtual memory on the server, the NetVault process nvrascontroller could not allocate enough memory to communicate with the DR containers, hence Media manager marking them offline and jobs failing.
-
Resolution
If an Exchange restore for RME is done using the NVBU server, ensure the recovery is done outside of the backup window.
Alternatively use an another server for the RME recovery to take the load off the NetVault Backup server which should be entirely dedicated to NetVault backup operations only.