The following errors were logged in the Binary Log, after the VSS Writers had been obtained and before the System State data could be backed up:
1] Error 2011/09/27 09:55:48 91 Jobs NVSERVER Job manager lost message channel unexpectedly
2] Error 2011/09/27 09:55:48 91 Jobs NVSERVER Fatal error: Backup Lost Message Channel Unexpectedly
3] Error 2011/09/27 09:55:48 91 Jobs NVSERVER Job Status: Backup Failed
From the NetVault Server Job Manager Trace files we could see the following errors:
nvjobmgr3087.log
2 JOBMGR ??? 1352 100 0 85346002395 Report job status 'Waiting for job options' to schedule manager
2 JOBMGR ??? 1352 62 0 85346002395 Plugin is now in state Waiting for job options
2 JOBMGR ??? 1352 81 0 85346283649 Get plugin options screen
2 JOBMGR ??? 1352 157 0 85346283649 Options screen is at 0069A708
2 JOBMGR ??? 1352 365 0 85443878505 JobMgrJobKeepAliveAction
2 JOBMGR ??? 1352 365 0 85543880011 JobMgrJobKeepAliveAction
0 NET ??? 1352 84 0 85548302028 Connection down
0 JOBMGR ??? 1352 447 0 85548302028 Plugin lost message channel unexpectedly
From the NetVault Client File System Trace files we could see the following errors:
nvfs286.log
0 VSSBACK ??? 2940 57 0 85501051632 Snapshot set created.
0 MEM ??? 2940 17 12 85542160218 Failed to allocate 948 byte: The access code is invalid.
0 MEM ??? 2940 17 12 85542160218 Failed to allocate 948 byte: The access code is invalid.
0 MEM ??? 2940 4 1 85542160218 Unable to allocate 928 bytes of memory
0 PROCESS ??? 2940 209 0 85542160218 Crash program
From the NetVault Server Job Manager Trace we can see that the connection was lost and at the same time from the NetVault Client File System Trace, there was a memory issue logged after the Snapshot set was created.
I would advise to check the following items when troubleshooting VSS issues, as memory fragmentation is a cause of VSS issues:
IMPORTANT: The below information is provided as a general guidance and Quest Software are not liable for any problem(s) that arise following Microsoft configuration changes.
You should seek Microsoft expert advice to perform any Microsoft related troubleshooting or configuration change, and we recommend opening a case with Microsoft.
1] Make sure the machine is patched with the latest Service Packs and VSS roll-up patches.
2] Check space: At least 30% free is needed for snapshots.
3] Memory fragmentation is a cause of VSS issues, so reboot if possible.
4] Verify VSS shadow copies are enabled for each of the drives to backup (My Computer > right click drive > Properties > Shadow Copies).
5] Are there any other backups or applications trying to schedule a snapshot at the same time?
6] Run CHKDSK on each drive with the /f switch to fix any errors on the disk.
7] Make the COM+ Event System Service to Automatic.
8] Make sure Volume Shadow Copy Service is set to automatic.
The customer mentioned that he had below 25% space left on the Hard Disk, so after clearing some space on the hard disk (has now above 30% free) and a reboot, the jobs are now working again.
