Exchange DAG backup hangs (4040570)

During the backups it is observed that on remote site that “Proxy Master on 'DagDR02' has been launched to backup/restore databases” at this point the backup hangs until the backup is aborted.  This can be seen in binary log extract below.

Looking at the trace extract from the DR dag member  “DagDR02” it is observed that PowerShell command  (exshell.psc1" -command ". {Get-ExchangeServer -identity "DagDR02" | fl edition,admindisplayversion}") fails on this node.

Checked to see if these commands would run locally on "DagDR02" and they failed as seen below.

check the communication of remote DAG member "DagDR02" to the NetVault server and the real dag member MBX01 and found that DNS between these nodes did not work.

Binary log extract:

Job Message      2016-08-10 12:03:48      384        Jobs      FXTNV1 Clustered Application DAG01 is running on real machine MBX01

Information       2016-08-10 12:04:39      384        Data Plugin        FXTEXMBX01              Server 'DagDR02' is selected for Backup/Restore

Job Message      2016-08-10 12:04:39      384        Data Plugin        FXTEXMBX01              Proxy Master on 'DagDR02' has been launched

Information       2016-08-10 12:04:39      384        Data Plugin        FXTEXMBX01              Proxy Master on 'DagDR02' has been launched to backup/restore databases

Job Message      2016-08-10 12:04:39      384        Data Plugin        FXTEXMBX01              Master will now wait for DAG processing to complete

Error     2016-08-10 12:09:29      384        Jobs      FXTNV1 Fatal error: Aborted by user

Trace extract:

2   EXCHUTIL   ???   7040   1079   0   120439145044   Found the Exchange Server Binaries path 'C:\Program Files\Microsoft\Exchange Server\V14\bin\'

2   EXCHUTIL   ???   7040   699   0   120439145044   Method ExchangeGetBinPath: ends, success = TRUE

2   EXCHUTIL   ???   7040   1074   0   120439145044   Execute the command 'powershell.exe -PSConsoleFile "C:\Program Files\Microsoft\Exchange Server\V14\bin\exshell.psc1" -command ". {Get-ExchangeServer -identity "DagDR02" | fl edition,admindisplayversion}" > 904380043800000000017040 2> 904380043800000000027040'

0   EXCHUTIL   ???   7040   953   1   120939144799   'powershell.exe -PSConsoleFile "C:\Program Files\Microsoft\Exchange Server\V14\bin\exshell.psc1" -command ". {Get-ExchangeServer -identity "DagDR02" | fl edition,admindisplayversion}" > 904380043800000000017040 2> 904380043800000000027040'  command failed with Reason: 'Timedout' Stopped after 300 seconds

2   EXCHUTIL   ???   7040   1079   0   120939144799   Found the Exchange Server Binaries path 'C:\Program Files\Microsoft\Exchange Server\V14\bin\'

2   EXCHUTIL   ???   7040   699   0   120939144799   Method ExchangeGetBinPath: ends, success = TRUE

2   EXCHUTIL   ???   7040   1074   0   120939144799   Execute the command 'powershell.exe -PSConsoleFile "C:\Program Files\Microsoft\Exchange Server\V14\bin\exshell.psc1" -command ". {Get-ExchangeServer -identity "dag01" | fl edition,admindisplayversion}" > 904380043800000000037040 2> 904380043800000000047040'

Power shell command failures:

PS C:\Windows\system32> Get-ExchangeServer -identity "DagDR02" | fl edition,admindisplayversion

Get-ExchangeServer : Value cannot be null.

Parameter name: serverSettings

At line:1 char:19

+ Get-ExchangeServer <<<< -identity "DagDR02" | fl edition,admindisplayversion

+ CategoryInfo : NotSpecified: (:) [Get-ExchangeServer], ArgumentNullException

+ FullyQualifiedErrorId : System.ArgumentNullException,Microsoft.Exchange.Management.SystemConfigurationTasks.GetE

xchangeServer

This is known cisco bug: CSCtj94130 see link below

http://icisystem.blogspot.co.uk/2015/06/cisco-notification-alert-nexus-5000.html

Permanent fix of applying static routing to the Nexus switch being used required.