-
Title
VSS: Backup fails with "The system was unable to flush I/O writes" -
Description
You experience backup failures with the following error displayed.
Stack trace example:
Replay.Agent.Contracts.Transfer.TransferFailedException: The transfer failed: 'There was a problem with the VSS subsystem on 'Agent01'.' ---> System.AggregateException: One or more errors occurred. ---> Replay.Core.Contracts.Transfer.VssGeneralErrorException: There was a problem with the VSS subsystem on 'Agent01'. ---> Replay.Agent.Contracts.ShadowCopy.ShadowCopyException: DoSnapshotSet failed with error: 'The system was unable to flush I/O writes. This can be a transient problem. It is recommended to wait ten minutes and try again, up to three times.'
-
Cause
This error maybe as a result of an issue with the VSS subsystem on the Agent machine or Hyper-V host caused by VSS not being able to quiesce data to the disk within the VSS timeout window. Potential environmental issues that may cause the problem are:
- Corruption of the NTFS structure of the disk
- High fragmentation of the disk (20%+)
- High IO on the disk
- No free space on the SRP and the agent change log is locked
NOTE: In Hyper-V environments, this might be due to Hyper-V integration services called Backup for Window Server 2012 R2 Hosts, and the way/speed that the OS releases VSS for the requester (in this case AppAssure software).
-
Resolution
Following a possible resolutions to mitigate against environmental issues that can be the cause of the problem:
- Corruption of the NTFS structure of the disk
- High IO on the disk
- No free space on the System Reseve Partition (SRP) and the agent change log is locked
- For Hyper-V environments, try disabling any snapshot by Hyper-V, if it is already enabled.
- Disk Misalignment
Corruption of the NTFS structure of the disk
- Run chkdsk to verify the integrity of the volume
- Open an administrative command prompt.
- Run "chkdsk /r ".
- Ensure that any errors that are found are fixed.
High fragmentation of the disk
- Run the defrag analysis tool on each of the disks. If the fragmentation is found to be greater than 10% then the disk should be defragmented. For disks that are low on free space, running disk defrag may be necessary even when the fragmentation is less than 10%.
High IO on the disk
If possible reduce the IO on the disk. If it is not possible to reduce the IO on the disk, then attempt to schedule around it.
- Check the current Storage utilization and performance and discard possible I/O bottlenecks or Split IO/Sec with Windows Perfmon.
- Set the backups to run at a low IO time:
- Select the agent in the Core Console and click on the Summary tab.
- Select the volume and click "Set Schedule"
- Adjust the interval or snapshot time so that each volume is scheduled to back up at different times.
- Stagger the Protection Intervals:
- Select the agent in the Core Console and click on the Summary tab.
- Select the volume and click "Set Schedule"
- Adjust the interval or snapshot time so that each volume is scheduled to back up at different times.
No free space on the System Reserve Partition (SRP) and the agent change log is locked
When the SRP is full, there is no free space on this volume for VSS to create a shadow copy. Furthermore, when the AppAssure Agent change log is locked (cannot take possession to delete the file), VSS will timeout as this log file cannot be accessed.
- To free space in the SRP remove the volume from protection schedule.
- Refresh metadata and wait a few minutes. This will remove the change logs.
- Check for free space on the server.
- When SRP has free space add it back to the protection schedule.
For Hyper-V environments, try disabling any snapshot by Hyper-V, if it is already enabled.
- Disable the backup feature for the VM. The setting is found on the Hyper V manger under integration services and un-check backup.
For VMware environments, please do not run any VMware queisced snapshots manually or using other backup software.
- Please do not run any VMware quiesced snapshots manually or using other backup software
Disk Misalignment
- Check for possible disk misalignment on the Agent's disks, more information here.
When all other factors were ruled out but VSS subsystem still won't create and clean a shadow copy, here a list of actions that may indicate which volume is causing such behavior- Right-click any disk on the protected machine and select "Configure Shadow Copies"
- Make sure that there are absolutely no shadow copies are listed under all volumes/disks. If there are, please delete them
- Check each volume's Settings and make sure that shadow copies are: set onto the same volume and has No Limit
- Attempt to create a shadow copy manually by clicking on Create Now. Delete the shadow copy once it was created successfully.
Failing volume with generate the "flush I/O" error. For such volume please open CMD in 'Run As Admin" mode and initiate a command:
chkdsk <volume>: /FPlease note that doing that on non-OS volumes will cause them to dismount. Doing that on the OS volumes will generate a message if chkdsk can be scheduled to run on the next reboot. In such case please accept that and reboot the machine, allowing chkdsk to process the OS volume.