Hyper-V Backups fail with VSS errors when backing up Cluster Shared Volumes (4035588)

Hyper-V Backups fail with VSS errors when backing up Cluster Shared Volumes

Hyper-V Plugin backups are failing with VSS errors, VSS errors are also visible on the failing VMs Event logs:

A VSS writer has rejected an event with error 0x800423f3, The writer experienced a transient error.  If the backup process is retried,
the error may not reoccur. Changes that the writer made to the writer components while handling the event will not be available to the requester. Check the event log for related events from the application hosting the VSS writer.

Operation:
  PrepareForSnapshot Event

Context:
  Execution Context: Writer
  Writer Class Id: {66841cd4-6ded-4f4b-8f17-fd23f8ddc3de}
  Writer Name: Microsoft Hyper-V VSS Writer
  Writer Instance ID: {a34d8ba2-a62a-4c02-aaac-9bbb954a106f}
  Command Line: C:\Windows\system32\vmms.exe

**The Hyper-V Writer also faults**

Writer name: 'Microsoft Hyper-V VSS Writer'
  Writer Id: {66841cd4-6ded-4f4b-8f17-fd23f8ddc3de}
  Writer Instance Id: {a34d8ba2-a62a-4c02-aaac-9bbb954a106f}
  State: [8] Failed
  Last error: Retryable error

VMs currently owned by different nodes which have files on a common Cluster Shared Volume cannot be backed up simultaneously when using the default system VSS provider. The second backup will fail as the shared volume will be owned by the first node. You can significantly reduce the amount of time that the first node owns the shared volume by using hardware snapshots through the VSS provider supplied by your storage hardware vendor. Once the hardware snapshot is completed, the backup process can be initiated from the second node, therefore the backups can be running concurrently from both nodes. To determine which node owns the volume, navigate the Windows menu items: ServerManager > Features > Failover Cluster Manager > cluster name > Cluster Shared Volumes. The Cluster Shared Volume windows displays the status of the volume, including ownership, and if a backup is in progress (with redirected I/O enabled).

 The following is taken from the NetVault Backup Plugin for Hyper-V 1.7 User's Guide, pg. 16

If you use the VSS backup method to support a failover cluster with CSVs, make sure that you create backup jobs that are scheduled to run at different times for the different nodes in the cluster. Only one node functions as the owner of the CSV at a time, so the jobs must run separately to prevent the backup from failing. If an overlap occurs (that is, the second job starts before the first job completes) and the first job fails, subsequent backups will also fail until you manually delete the folder with the corresponding VM's name from the Working Directory.