Windows 2008 System State backup fails on System files - "Failed to add extra items to the select (4042589)

When performing a System State backup on Windows 2008, the job fails with the following log messages:

Error 2009/02/02 17:11:40 34 Data Plugin NETVAULT Failed to add extra items to the selection tree
Severe Error 2009/02/02 17:11:40 34 Data Plugin NETVAULT Failed to prepare the selection tree and backup options

The File System trace shows problems with WFP (Windows Files Protector) files, also known as WRP (Windows Resource Protector) on Windows 2008:

0 SSFILES :2552 2 0 135456 Failed to add WFP files when creating files tree
0 NVFSWIN :2552 1001 0 135456 Failed to get system files tree from ss module : 'Failed to add protected files to the backup tree'
0 NVFSWIN :2552 1044 0 135456 Failed to add System State system files to extra items tree
0 HBBACKUP:2552 118 0 135456 Failed to get extra sub level backup items

The Windows Event Logs would in some cases show:
Cryptographic Services Failed while processing the OnIdentity() call in the System Writer Object.
Details: AddCoreCsiFiles: BeginFileEnumeration()Failed
SystemError: Access is Denied.
 

In order for the NetVault File System plugin to backup the System State, it relies relevant VSS writers.
Not to be confused with the NetVault VSS license.
To be able to enumerate the System Files on Windows 2008, we call the VSS System Writer, dependent on VSSUTIL.
There are several reasons for this problem to happen, as many variables are involved:

• In some cases the VSS System writer could either be missing, modified or in an unstable state.
• Also it could be a permissions issue that prevents us from enumerating or accessing those files.
• Related to a known Microsoft issue regarding the installation of MSSQL Server 2005/8 or Exchange 2007. These applications seem to modify the COM+ Catalog and removes the need to use the original 'System Writer', installing their own writers. Any backup solution requesting the system files after this will fail as the needed VSS System Writer will be missing and the wrong writer used. This is not a problem with NetVault:Backup but rather a Microsoft issue encountered by other backup solutions.

There are several areas to check:
 

1 - Is there a VSS System Writer?
In the command prompt run: 'vssadmin list writers' . This will list of all VSS writers.
It should have a state of [stable] 1.

2 - Was MSSQL 2005/8 or Exchange 2007 installed recently?
If so then it would force the system to use the new SQL VSS Writer instead of the original MSDEWriter for backups which will be removed.
You can disable 'MSDEVersionChecking' by editing the Registry. Then reboot the machine.
IMPORTANT: BakBone Software Ltd is not responsible nor liable for any problem arising from editing the Windows Registry.
We recommend you contact Microsoft for expert advice on this issue.

3- In the event logs show a "SystemError: Access is Denied." when the enumeration of the System Files begins, it could be a permissions issue.
In this case please refer to the following Microsoft solution:
http://social.technet.microsoft.com/Forums/en-US/windowsbackup/thread/b5236028-4bfe-400c-90e2-b6d40b08e776
a- Overwrite permissions as follows:
Takeown /f %windir%\winsxs\filemaps\* /a
icacls %windir%\winsxs\filemaps\*.* /grant "NT AUTHORITY\SYSTEM:(RX)"
icacls %windir%\winsxs\filemaps\*.* /grant "NT Service\trustedinstaller:(F)"
icacls %windir%\winsxs\filemaps\*.* /grant BUILTIN\Users:(RX)
b- Stopped and started Cryptographic service
net stop cryptsvc
net start cryptsvc
c- Verify the "System Writer" shows up in list of writers using "vssadmin list writers"
This will allow for the backups to complete.


4- Finally, there is a workaround in NetVault:
If backing up the 'System Files' using the VSS System Writer is impossible, you can still back them up by selecting the whole C:\ drive.
The majority of system files are under C:\WINDOWS\system32\ but not all.
Essentially if the Selection Set excludes the System Files from the System State node, and the rest of the Fixed Drive is selected, we can be sure that all System Files will be included in the backup and allow for a successful DR scenario.
This way all the items needed for a consistent System State backup will be included in the same saveset.
The use of the NetVault VSS license is highly advisable to make sure open files are backed up and not skipped.

IMPORTANT:
This is not a limitation of the NetVault File System Plugin but rather a Microsoft issue.